|
Replies:
33
-
Last Post:
Sep 11, 2008 7:36 AM
by: join2008
|
Threads:
[
Previous
|
Next
]
|
|
Posts:
45
From:
Registered:
3/9/05
|
|
|
|
File events notification mechansim
Posted:
May 3, 2006 2:44 PM
|
|
Hi All,
We are working on adding a 'file events notification/monitoring' mechanism
to the Event ports framework in Solaris.
There are several different implementations around this file events
notification mechanism currently available, like 'dnotify, inotify' for
Linux or the 'FAM' from SGI etc.
I am interested in knowing what are the file event types that people
care about/find useful, any use cases. Also please share your experience
if you have used any of the file event notification facilities currently
available.
Here are a some of the file event types considered.
FILE_OPEN File opened.
FILE_CLOSE File closed.
FILE_WRITE File was written to.
FILE_READ File was read.
FILE_ATTRIB_CHANGE File attributes changed
FILE_CREATE File/Directory was created.
FILE_REMOVE/UNLINK File/Directory was removed/deleted.
FILE_RENAME_TO/FROM File was renamed to/from.
Expected behavior...
Once any event is delivered should it get de-registered automatically or
not? If it gets de-registered, the application will have to register it
again before it can receive more events on that file. If it does not
get de-registered, it will continue to receive events as they occur
until it is explicitly de-registered.
If a file, being monitored, gets removed/unlinked should it get
De-registered(stop monitoring) or should it continue to be monitored
and deliver events? The file may still be open and being accessed.
This facility will be added to the Event ports framework.
The Event ports framework, introduced in Solaris 10, is an unified event
notification framework. The 'file events notification facility' will
be added as a new event source to the Event ports framework.
If you like to know more about Event ports you can read about it here.
You will see some examples using Event ports.
http://developers.sun.com/solaris/articles/event_completion.html
http://blogs.sun.com/roller/page/barts/20040720
Here it is how the interfaces for 'file event notification mechanism',
would be.
int userdata;
port_event_t pe;
char *fname[]= "/var/tmp/testfile"
/*
* create the port on which to receive the
* events. This port can multiplex events from
* other event sources too.
*/
port = port_create();
/*
* associate or register the file events for a file on this
* port.
*/
port_associate(port, PORT_SOURCE_FILE, (uintptr_t)fname,
FILE_OPEN|FILE_ATTRIB, &userdata);
/*
* collect events from the port
*/
port_getn(port, &pe);
/*
* De-register/stop monitoring.
*/
port_dissociate(port, PORT_SOURCE_FILE, (uintptr_t)fname);
The 'userdata' pointer passed in is delivered along with the event in
the port_event_t structure.
The object 'fname' is the name of the file which is to be monitored.
Instead, the object could be a structure with some additional
info to be passed in/returned back.
Thanks,
-Prakash.
_______________________________________________
perf-discuss mailing list
perf-discuss at opensolaris dot org
|
|
|
Posts:
850
From:
GB
Registered:
3/9/05
|
|
|
|
Re: File events notification mechansim
Posted:
May 3, 2006 5:19 PM
in response to: praks
|
|
On Wed, May 03, 2006 at 02:44:01PM -0700, Prakash Sangappa wrote:
> FILE_CREATE File/Directory was created.
I think GNOME and others want to be able to watch an entire directory tree. It
sounds like the proposal is that you can only watch a particular path
"/etc/non_existent_file" for FILE_CREATE? If not, can you expand on this? What
do I have to pass in as the 'object' to port_associate() with these events?
Parent directory? If so, is it recursive?
How does it interact with unmounting the underlying filesystem?
You say that list is "some of the event types" planned. Presumably there are
others?
regards
john
_______________________________________________
perf-discuss mailing list
perf-discuss at opensolaris dot org
|
|
|
|
|
Posts:
45
From:
Registered:
3/9/05
|
|
|
|
Re: File events notification mechansim
Posted:
May 3, 2006 7:31 PM
in response to: johnlev
|
|
John Levon wrote:
> On Wed, May 03, 2006 at 02:44:01PM -0700, Prakash Sangappa wrote:
>
>
>> FILE_CREATE File/Directory was created.
>>
>
> I think GNOME and others want to be able to watch an entire directory tree. It
> sounds like the proposal is that you can only watch a particular path
> "/etc/non_existent_file" for FILE_CREATE? If not, can you expand on this? What
> do I have to pass in as the 'object' to port_associate() with these events?
> Parent directory? If so, is it recursive?
>
>
If you where to watch for events on an entire directory tree, what types
of events that
would be? How would this be useful?
Yes, it is to watch a file or a directory. The events which are
relevant will be delivered.
So the case where you want to watch for a file being created
"/etc/non_existent_file"
can be to just a watch on the directory "/etc". The FILE_CREATE event
would mean
that a file or a directory was created under "/etc". Once the
application gets this event,
it can go and check if the file got created.
We could make it watch for a particular file being created like
"/etc/non_existent".
But once this event is delivered, the application will have to again
check to see if this
file exists as it can get deleted by the time the event it received.
Clearly this is
going to be racy.
Therefore the event types can be simplified. Instead of having events
for OPEN, READ, WRITE, CREATE we could just have
ACCESS & MODIFY. Then you just watch to see if a file or
a directory is accessed or modified and then perform the necessary
checks and take action.
> How does it interact with unmounting the underlying filesystem?
>
It will hold a reference to the 'vnode' of the file/directory in the
kernel, while we have
the file events monitor on it. If the file system gets unmounted, it
will de-register the
monitor and send an exception event saying the filesystem got unmounted.
These exception events need to be defined.
> You say that list is "some of the event types" planned. Presumably there are
> others?
>
>
This was an initial set to get the discussion going. The idea is to keep
it simpler and
limit them to a basic set of events that will server the purpose.
Rgds,
-Prakash.
> regards
> john
> _______________________________________________
> perf-discuss mailing list
> perf-discuss at opensolaris dot org
>
_______________________________________________
perf-discuss mailing list
perf-discuss at opensolaris dot org
|
|
|
|
|
Posts:
850
From:
GB
Registered:
3/9/05
|
|
|
|
Re: File events notification mechansim
Posted:
May 3, 2006 8:23 PM
in response to: praks
|
|
On Wed, May 03, 2006 at 07:31:08PM -0700, Prakash Sangappa wrote:
> If you where to watch for events on an entire directory tree, what types
> of events that would be?
Presumably there would be some way to specify, but file creation/deletion would
be the most obviously useful events.
> How would this be useful?
Beagle wants it, I think:
http://beaglewiki.org/Main_Page
as well as some other GNOME things, apparently.
> can be to just a watch on the directory "/etc". The FILE_CREATE event would
> mean that a file or a directory was created under "/etc". Once the
> application gets this event, it can go and check if the file got created.
Does this mean we don't get told /what/ got created? Is an application that
wants to know "what files are disappearing/appearing under /foo/bar/?" going to
have to readdir() the whole directory every time it gets an event?
> >How does it interact with unmounting the underlying filesystem?
>
> It will hold a reference to the 'vnode' of the file/directory in the kernel,
> while we have the file events monitor on it. If the file system gets
> unmounted, it will de-register the monitor and send an exception event saying
> the filesystem got unmounted. These exception events need to be defined.
Sounds sensible...
regards
john
_______________________________________________
perf-discuss mailing list
perf-discuss at opensolaris dot org
|
|
|
|
|
Posts:
1,172
From:
US
Registered:
3/9/05
|
|
|
|
Re: File events notification mechansim
Posted:
May 4, 2006 8:46 AM
in response to: johnlev
|
|
John Levon wrote:
> On Wed, May 03, 2006 at 07:31:08PM -0700, Prakash Sangappa wrote:
>
>> If you where to watch for events on an entire directory tree, what types
>> of events that would be?
>
> Presumably there would be some way to specify, but file creation/deletion would
> be the most obviously useful events.
>
>> How would this be useful?
>
> Beagle wants it, I think:
>
> http://beaglewiki.org/Main_Page
>
> as well as some other GNOME things, apparently.
>
This cannot scale. I cannot expect a single application to able to
receive events for all modifications to a 10 TB filesystem in heavy use.
Clearly, the app will need a backup plan. And if this is a search app,
what about file modifications?
>> can be to just a watch on the directory "/etc". The FILE_CREATE event would
>> mean that a file or a directory was created under "/etc". Once the
>> application gets this event, it can go and check if the file got created.
>
> Does this mean we don't get told /what/ got created? Is an application that
> wants to know "what files are disappearing/appearing under /foo/bar/?" going to
> have to readdir() the whole directory every time it gets an event?
>
Otherwise we get into the queued event problem; what happens if the
application is watching a directory w/ a million files, and someone does
rm * in that directory? Do we generate a million events? Clearly there
are limits to the number of events we can queue in the kernel, esp.
since the application isn't obligated to read them in a timely fashion.
How about the file modification event? Do we generate them on each
write? On each read?
I'd rather have a model like signals; multiple file writes are combined
into one event until that event is read by the application; any
subsequent writes generate another event. This would work equally well
for directories.
I see this as very useful to avoid the {sleep(); stat() } loops we often
see. It's not a mechanism to insert an application as an synchronous
interposer into the filesystem VOPS. I don't know how to do this w/o
having each application write code to cope with the case where the
kernel generates events faster than the application can handle them.
A file manager such as Nautilus could use this to avoid the following
loops:
/1: pollsys(0x08270888, 5, 0x08047058, 0x00000000) = 0
/1: xstat(2, "/home/barts/Desktop", 0x08046F08) = 0
/1: ioctl(4, I_NREAD, 0x08046FD0) = 0
/2: pollsys(0x0843FCA0, 10, 0x00000000, 0x00000000) (sleeping...)
/1: pollsys(0x08270888, 5, 0x08047058, 0x00000000) (sleeping...)
/1: pollsys(0x08270888, 5, 0x08047058, 0x00000000) = 0
/1: xstat(2, "/home/barts/Desktop", 0x08046F08) = 0
The nscd could use this to watch for modifications to configuration
files rather than stat'ing them before each cache lookup.
The model of watching all events in a filesystem hierarchy would
seem also to have significant security issues; can an application
use this to watch the creation of files in someone else's 0700
directory?
- Bart
--
Bart Smaalders Solaris Kernel Performance
barts at cyber dot eng dot sun dot com http://blogs.sun.com/barts
_______________________________________________
perf-discuss mailing list
perf-discuss at opensolaris dot org
|
|
|
|
|
Posts:
850
From:
GB
Registered:
3/9/05
|
|
|
|
Re: File events notification mechansim
Posted:
May 4, 2006 11:08 AM
in response to: barts
|
|
On Thu, May 04, 2006 at 08:46:01AM -0700, Bart Smaalders wrote:
> >http://beaglewiki.org/Main_Page
> >
> >as well as some other GNOME things, apparently.
>
> This cannot scale.
It doesn't necessarily need to scale; from my understanding, Beagle is more
about watching ~/Documents/ than big lots of data.
> >Does this mean we don't get told /what/ got created? Is an application that
> >wants to know "what files are disappearing/appearing under /foo/bar/?" going
> >to have to readdir() the whole directory every time it gets an event?
>
> Otherwise we get into the queued event problem; what happens if the
> application is watching a directory w/ a million files, and someone does
> rm * in that directory? Do we generate a million events? Clearly there
> are limits to the number of events we can queue in the kernel, esp.
> since the application isn't obligated to read them in a timely fashion.
Forcing a (recursive!) readdir() every time can't scale either; it just pushes
the problem out all the userspace apps. Perhaps a compromise approach would
work, so at least the readdir() cost is amortized; i.e. give names up to a
particular limit.
Or how do you expect Beagle to be able to work nicely? Is this just going to
remain something explicitly unsupportable?
> I'd rather have a model like signals; multiple file writes are combined
> into one event until that event is read by the application; any
> subsequent writes generate another event.
Would work fine for modifications, yes.
> I see this as very useful to avoid the {sleep(); stat() } loops we often
> see. It's not a mechanism to insert an application as an synchronous
> interposer into the filesystem VOPS.
I wasn't trying to suggest it was. Synchronisation is neither needed nor
wanted.
> The nscd could use this to watch for modifications to configuration
> files rather than stat'ing them before each cache lookup.
I wasn't suggesting that a non-recursive approach doesn't solve a whole class
of such situations; it does. In fact, I was merely trying to raise awareness of
what applications like Beagle actually need in terms of notifications. If it's
really too hard to do, that's a pity.
regards,
john
_______________________________________________
perf-discuss mailing list
perf-discuss at opensolaris dot org
|
|
|
|
|
Posts:
45
From:
Registered:
3/9/05
|
|
|
|
Re: File events notification mechansim
Posted:
May 4, 2006 12:52 PM
in response to: johnlev
|
|
John Levon wrote:
> On Thu, May 04, 2006 at 08:46:01AM -0700, Bart Smaalders wrote:
>
>>> Does this mean we don't get told /what/ got created? Is an application that
>>> wants to know "what files are disappearing/appearing under /foo/bar/?" going
>>> to have to readdir() the whole directory every time it gets an event?
>>>
>> Otherwise we get into the queued event problem; what happens if the
>> application is watching a directory w/ a million files, and someone does
>> rm * in that directory? Do we generate a million events? Clearly there
>> are limits to the number of events we can queue in the kernel, esp.
>> since the application isn't obligated to read them in a timely fashion.
>>
>
> Forcing a (recursive!) readdir() every time can't scale either; it just pushes
> the problem out all the userspace apps. Perhaps a compromise approach would
> work, so at least the readdir() cost is amortized; i.e. give names up to a
> particular limit.
>
> Or how do you expect Beagle to be able to work nicely? Is this just going to
> remain something explicitly unsupportable?
>
>
I am not sure if 'inotify' can watch an entire directory tree? Is
Beagle using inotify
currently to watch directory tree? What are Beagle's requirements? May
be we should
understand that better.
Like Bart mentioned how does it resolve access permissions along the path?
What about nested mount points?
If the events have to be queued, then we will have to set the size(# of
events)
of the queue along with the watch request. Would this address the needs
completely?
Implementing this would complicate the interfaces and probably open up
more issues.
>> I'd rather have a model like signals; multiple file writes are combined
>> into one event until that event is read by the application; any
>> subsequent writes generate another event.
>>
>
> Would work fine for modifications, yes.
>
We could have this for all the event types we support. If the event is
set in the
notification, it would mean that event has occurred since the last time
it was
checked. The application has to check/verify it once it receives the event.
With this approach there will not be a need for queuing events.
-Prakash.
>> I see this as very useful to avoid the {sleep(); stat() } loops we often
>> see. It's not a mechanism to insert an application as an synchronous
>> interposer into the filesystem VOPS.
>>
>
> I wasn't trying to suggest it was. Synchronisation is neither needed nor
> wanted.
>
>
>> The nscd could use this to watch for modifications to configuration
>> files rather than stat'ing them before each cache lookup.
>>
>
> I wasn't suggesting that a non-recursive approach doesn't solve a whole class
> of such situations; it does. In fact, I was merely trying to raise awareness of
> what applications like Beagle actually need in terms of notifications. If it's
> really too hard to do, that's a pity.
>
> regards,
> john
> _______________________________________________
> perf-discuss mailing list
> perf-discuss at opensolaris dot org
>
_______________________________________________
perf-discuss mailing list
perf-discuss at opensolaris dot org
|
|
|
|
|
Posts:
1,901
From:
NZ
Registered:
6/16/05
|
|
|
|
Re: File events notification mechansim
Posted:
May 4, 2006 3:43 PM
in response to: johnlev
|
|
Hey,
John Levon wrote:
> On Thu, May 04, 2006 at 08:46:01AM -0700, Bart Smaalders wrote:
>
>>> http://beaglewiki.org/Main_Page
>>>
>>> as well as some other GNOME things, apparently.
>> This cannot scale.
>
> It doesn't necessarily need to scale; from my understanding, Beagle is more
> about watching ~/Documents/ than big lots of data.
Yeah, currently Beagle only indexes a relatively small amount of
per-user data, generally in $HOME - however, as has been mentioned, it's
probably one of the first proper use cases of inotify.
I'd suggest that it's definitely worth looking at what inotify does -
given that there seems to have been a lot of churn with
dnotify/inotify/FAM/gamin/ etc.., there's probably some implementation
lessons to be learned from our Linux neighbours [some of
http://kerneltrap.org/node/3847 might be interesting reading].
FILE_CLOSE_WRITE is missing from the list of events posted previously
that is apparently useful, as suggested by some GNOME developers.
Glynn
Glynn
>>> Does this mean we don't get told /what/ got created? Is an application that
>>> wants to know "what files are disappearing/appearing under /foo/bar/?" going
>>> to have to readdir() the whole directory every time it gets an event?
>> Otherwise we get into the queued event problem; what happens if the
>> application is watching a directory w/ a million files, and someone does
>> rm * in that directory? Do we generate a million events? Clearly there
>> are limits to the number of events we can queue in the kernel, esp.
>> since the application isn't obligated to read them in a timely fashion.
>
> Forcing a (recursive!) readdir() every time can't scale either; it just pushes
> the problem out all the userspace apps. Perhaps a compromise approach would
> work, so at least the readdir() cost is amortized; i.e. give names up to a
> particular limit.
>
> Or how do you expect Beagle to be able to work nicely? Is this just going to
> remain something explicitly unsupportable?
>
>> I'd rather have a model like signals; multiple file writes are combined
>> into one event until that event is read by the application; any
>> subsequent writes generate another event.
>
> Would work fine for modifications, yes.
>
>> I see this as very useful to avoid the {sleep(); stat() } loops we often
>> see. It's not a mechanism to insert an application as an synchronous
>> interposer into the filesystem VOPS.
>
> I wasn't trying to suggest it was. Synchronisation is neither needed nor
> wanted.
>
>> The nscd could use this to watch for modifications to configuration
>> files rather than stat'ing them before each cache lookup.
>
> I wasn't suggesting that a non-recursive approach doesn't solve a whole class
> of such situations; it does. In fact, I was merely trying to raise awareness of
> what applications like Beagle actually need in terms of notifications. If it's
> really too hard to do, that's a pity.
>
> regards,
> john
> _______________________________________________
> perf-discuss mailing list
> perf-discuss at opensolaris dot org
_______________________________________________
perf-discuss mailing list
perf-discuss at opensolaris dot org
|
|
|
|
|
Posts:
45
From:
Registered:
3/9/05
|
|
|
|
Re: File events notification mechansim
Posted:
May 4, 2006 5:11 PM
in response to: gman
|
|
Glynn Foster wrote:
>
> Yeah, currently Beagle only indexes a relatively small amount of
> per-user data, generally in $HOME - however, as has been mentioned,
> it's probably one of the first proper use cases of inotify.
>
> I'd suggest that it's definitely worth looking at what inotify does -
> given that there seems to have been a lot of churn with
> dnotify/inotify/FAM/gamin/ etc.., there's probably some implementation
> lessons to be learned from our Linux neighbours [some of
> http://kerneltrap.org/node/3847 might be interesting reading].
>
> FILE_CLOSE_WRITE is missing from the list of events posted previously
> that is apparently useful, as suggested by some GNOME developers.
>
Yes, I have looked at some of the issue with 'dnotify' that got addressed
by 'inotify' and also some of the issue with inotify.
Here is another pointer which lists some issues with 'inotify'. These
may have
been addressed already with the later version of inotify.
http://manic.desrt.ca/inotify
Proposed interfaces:
- Unlike the 'inotify' interfaces, which uses 'ioctls' to a device, our
interfaces will be based on the existing Event ports framework. The
events
will be delivered to a specified 'Event port'(which is an fd).
The Event port can receive events from multiple sources. Currently
available event sources in Solaris 'poll, aio, timer, user,
message queue(this was recently integrated)'.
More event sources can be added.
Ex,
if the application needs to wait on a 'poll' event and also receive
file events
notifications, it can receive both these types of events on one Event
port.
- The application does not have to open the file/directory being watched.
Therefore it will not prevent the file from being deleted or
unmounting the filesystem. These where issues with 'dnotify'.
- If the file system gets unmounted, it will automatically de-register
the event notification and send an event indicating that.
It can also send a 'file deleted event' when watching a file which
gets deleted.
- A pointer to some user data can be passed in when registering file
event notication.
The user data will be returned along with the event notification.
This was something
missing with the inotify interface.
I think 'inotify' returns something called 'watch descriptors' when
registering the watches.
They had some issue with 'watch descriptors' being reused that would
cause confusion in
identifying the received events. This issue may have been addressed
already with their
new version.
This problem should not exist with the event ports interface since a
user data pointer will
be returned with the event. The application can differentiate the
events based on that.
- To de-register or to re-register, the object(file) needs to be passed
again. But the
file name could disappear(get removed) from the directory. To
accommodate this,
we could change the passed in object to a structure which will
include the filename.
Once it gets registered, the system can return an unique id. So
subsequent actions
(de-register, re-register), we depend on this 'id'.
Ex.
file_event {
uintptr_t id; /* id returned after registering */
int len; /* length of the file name */
char fname[0]; /* filename */
}fobj;
So, now the interface will be
port_associate(port, PORT_SOURCE_FILE, (uintptr_t)&fobj, ... )
So this 'fobj' structure can be passed to de-register(port_dissociate)
or re-register(port_associate again) the file events.
FILE_CLOSE_WRITE - What is the purpose of this event? If it is
found to be useful, we could include it.
Rgds,
-Prakash.
>
>
> Glynn
>
>
> Glynn
>
>>>> Does this mean we don't get told /what/ got created? Is an
>>>> application that
>>>> wants to know "what files are disappearing/appearing under
>>>> /foo/bar/?" going
>>>> to have to readdir() the whole directory every time it gets an event?
>>> Otherwise we get into the queued event problem; what happens if the
>>> application is watching a directory w/ a million files, and someone
>>> does
>>> rm * in that directory? Do we generate a million events? Clearly
>>> there
>>> are limits to the number of events we can queue in the kernel, esp.
>>> since the application isn't obligated to read them in a timely fashion.
>>
>> Forcing a (recursive!) readdir() every time can't scale either; it
>> just pushes
>> the problem out all the userspace apps. Perhaps a compromise approach
>> would
>> work, so at least the readdir() cost is amortized; i.e. give names up
>> to a
>> particular limit.
>>
>> Or how do you expect Beagle to be able to work nicely? Is this just
>> going to
>> remain something explicitly unsupportable?
>>
>>> I'd rather have a model like signals; multiple file writes are
>>> combined into one event until that event is read by the application;
>>> any subsequent writes generate another event.
>>
>> Would work fine for modifications, yes.
>>
>>> I see this as very useful to avoid the {sleep(); stat() } loops we
>>> often
>>> see. It's not a mechanism to insert an application as an synchronous
>>> interposer into the filesystem VOPS.
>>
>> I wasn't trying to suggest it was. Synchronisation is neither needed nor
>> wanted.
>>
>>> The nscd could use this to watch for modifications to configuration
>>> files rather than stat'ing them before each cache lookup.
>>
>> I wasn't suggesting that a non-recursive approach doesn't solve a
>> whole class
>> of such situations; it does. In fact, I was merely trying to raise
>> awareness of
>> what applications like Beagle actually need in terms of
>> notifications. If it's
>> really too hard to do, that's a pity.
>>
>> regards,
>> john
>> _______________________________________________
>> perf-discuss mailing list
>> perf-discuss at opensolaris dot org
> _______________________________________________
> perf-discuss mailing list
> perf-discuss at opensolaris dot org
_______________________________________________
perf-discuss mailing list
perf-discuss at opensolaris dot org
|
|
|
|
|
Posts:
1,901
From:
NZ
Registered:
6/16/05
|
|
|
|
Re: File events notification mechansim
Posted:
May 4, 2006 5:43 PM
in response to: praks
|
|
Hi,
Prakash Sangappa wrote:
[snip lots of good discussion that I can't really add to]
> FILE_CLOSE_WRITE - What is the purpose of this event? If it is
> found to be useful, we could include it.
I would think it would be most useful for the indexing system for search
ie. only index a file that has been written to *and* closed.
Glynn
_______________________________________________
perf-discuss mailing list
perf-discuss at opensolaris dot org
|
|
|
|
|
Posts:
5,504
From:
US
Registered:
3/9/05
|
|
|
|
Re: File events notification mechansim
Posted:
May 4, 2006 5:47 PM
in response to: praks
|
|
Prakash Sangappa wrote:
> - Unlike the 'inotify' interfaces, which uses 'ioctls' to a device, our
> interfaces will be based on the existing Event ports framework. The
> events
> will be delivered to a specified 'Event port'(which is an fd).
Can this fd be polled via poll/select or will all consumers of this interface
have to re-write their existing app logic to add another thread that sits in
port_get() all the time and then wakes up the main event loop that uses
poll/select instead?
--
-Alan Coopersmith- alan dot coopersmith at sun dot com
Sun Microsystems, Inc. - X Window System Engineering
_______________________________________________
perf-discuss mailing list
perf-discuss at opensolaris dot org
|
|
|
|
|
Posts:
370
From:
San Francisco, CA
Registered:
3/9/05
|
|
|
|
Re: File events notification mechansim
Posted:
May 4, 2006 5:50 PM
in response to: alanc
|
|
On Thu, May 04, 2006 at 05:47:06PM -0700, Alan Coopersmith wrote:
> Prakash Sangappa wrote:
> >- Unlike the 'inotify' interfaces, which uses 'ioctls' to a device, our
> > interfaces will be based on the existing Event ports framework. The
> >events
> > will be delivered to a specified 'Event port'(which is an fd).
>
> Can this fd be polled via poll/select or will all consumers of this
> interface
> have to re-write their existing app logic to add another thread that sits in
> port_get() all the time and then wakes up the main event loop that uses
> poll/select instead?
I believe the idea is more that you convert your pool/select loop into
a port_getn() on a single port, which all of your events are set up on.
No need for another thread.
Cheers,
- jonathan
--
Jonathan Adams, Solaris Kernel Development
_______________________________________________
perf-discuss mailing list
perf-discuss at opensolaris dot org
|
|
|
|
|
Posts:
5,504
From:
US
Registered:
3/9/05
|
|
|
|
Re: File events notification mechansim
Posted:
May 4, 2006 5:53 PM
in response to: jwadams
|
|
Jonathan Adams wrote:
> On Thu, May 04, 2006 at 05:47:06PM -0700, Alan Coopersmith wrote:
> I believe the idea is more that you convert your pool/select loop into
> a port_getn() on a single port, which all of your events are set up on.
Not so easy to do when the poll/select loop is buried deep in the core
of your GUI toolkit, as it is in most GUI apps.
--
-Alan Coopersmith- alan dot coopersmith at sun dot com
Sun Microsystems, Inc. - X Window System Engineering
_______________________________________________
perf-discuss mailing list
perf-discuss at opensolaris dot org
|
|
|
|
|
Posts:
370
From:
San Francisco, CA
Registered:
3/9/05
|
|
|
|
Re: File events notification mechansim
Posted:
May 4, 2006 5:58 PM
in response to: alanc
|
|
On Thu, May 04, 2006 at 05:53:51PM -0700, Alan Coopersmith wrote:
> Jonathan Adams wrote:
> >On Thu, May 04, 2006 at 05:47:06PM -0700, Alan Coopersmith wrote:
> >I believe the idea is more that you convert your pool/select loop into
> >a port_getn() on a single port, which all of your events are set up on.
>
> Not so easy to do when the poll/select loop is buried deep in the core
> of your GUI toolkit, as it is in most GUI apps.
Ah, but you can just poll/select on the port fd to get notifications there
are events on it. So you're fine.
Cheers,
- jonathan
--
Jonathan Adams, Solaris Kernel Development
_______________________________________________
perf-discuss mailing list
perf-discuss at opensolaris dot org
|
|
|
|
|
Posts:
5,504
From:
US
Registered:
3/9/05
|
|
|
|
Re: File events notification mechansim
Posted:
May 4, 2006 6:03 PM
in response to: jwadams
|
|
Jonathan Adams wrote:
> Ah, but you can just poll/select on the port fd to get notifications there
> are events on it. So you're fine.
Great. (I'd heard a bit about event ports, but this is the first use of them
that's sounded useful to anything close to what I work on, so I've never learned
their details.)
--
-Alan Coopersmith- alan dot coopersmith at sun dot com
Sun Microsystems, Inc. - X Window System Engineering
_______________________________________________
perf-discuss mailing list
perf-discuss at opensolaris dot org
|
|
|
|
|
Posts:
390
From:
IE
Registered:
6/17/05
|
|
|
|
Re: File events notification mechansim
Posted:
May 5, 2006 4:04 AM
in response to: praks
|
|
Hi Prakash,
I don't think it's the implementation that bridges between the kernel
and the user spaces that's important to JDS, and probably most other
people - it's the ultimate API that people will have to write to, and I
think from this perspective that sysevents is not what we want - it may
be part of the implementation of the final API, but it shouldn't really
be the main API that developers have to write to. At the moment there
are two projects, both with the same ABI (AFAIK), that are prevalent in
the Linux developer community:
* libfam - http://oss.sgi.com/projects/fam/
* gamin - http://www.gnome.org/~veillard/gamin/index.html
They are compared (with some bias of course) at:
http://www.gnome.org/~veillard/gamin/differences.html
It's an API like these that we need, so I think any solution proposed
(albeit sysevents underneath) should expose one of these APIs to the
user, if the same people produce this API as those that expose things
via sysevents, then things should work at their optimal. Whether it's
the SAME API is up for debate, but what ever we expose should at least
provide the same functionality.
One think that people have mentioned before as well is the handling of
distributed file-systems - how do you propose we handle these - this is
especially important on Solaris given that the majority of our customers
use NFS for their home directories. From what I understand of the File
Event Mechanism (FEM) in the kernel, we are using this in NFSv4, can
this provide us with the ability to see changes to files that occur on a
NFSv4 filesystem mounted on my desktop, for instance?
Thanks,
Darren.
Prakash Sangappa wrote:
> Glynn Foster wrote:
>>
>> Yeah, currently Beagle only indexes a relatively small amount of
>> per-user data, generally in $HOME - however, as has been mentioned,
>> it's probably one of the first proper use cases of inotify.
>>
>> I'd suggest that it's definitely worth looking at what inotify does -
>> given that there seems to have been a lot of churn with
>> dnotify/inotify/FAM/gamin/ etc.., there's probably some
>> implementation lessons to be learned from our Linux neighbours [some
>> of http://kerneltrap.org/node/3847 might be interesting reading].
>>
>> FILE_CLOSE_WRITE is missing from the list of events posted previously
>> that is apparently useful, as suggested by some GNOME developers.
>>
> Yes, I have looked at some of the issue with 'dnotify' that got addressed
> by 'inotify' and also some of the issue with inotify.
>
> Here is another pointer which lists some issues with 'inotify'. These
> may have
> been addressed already with the later version of inotify.
> http://manic.desrt.ca/inotify
>
>
> Proposed interfaces:
>
> - Unlike the 'inotify' interfaces, which uses 'ioctls' to a device, our
> interfaces will be based on the existing Event ports framework. The
> events
> will be delivered to a specified 'Event port'(which is an fd).
>
> The Event port can receive events from multiple sources. Currently
> available event sources in Solaris 'poll, aio, timer, user,
> message queue(this was recently integrated)'.
>
> More event sources can be added.
>
> Ex,
> if the application needs to wait on a 'poll' event and also receive
> file events
> notifications, it can receive both these types of events on one
> Event port.
>
> - The application does not have to open the file/directory being watched.
> Therefore it will not prevent the file from being deleted or
> unmounting the filesystem. These where issues with 'dnotify'.
>
> - If the file system gets unmounted, it will automatically de-register
> the event notification and send an event indicating that.
> It can also send a 'file deleted event' when watching a file which
> gets deleted.
>
> - A pointer to some user data can be passed in when registering file
> event notication.
> The user data will be returned along with the event notification.
> This was something
> missing with the inotify interface.
>
> I think 'inotify' returns something called 'watch descriptors' when
> registering the watches.
> They had some issue with 'watch descriptors' being reused that would
> cause confusion in
> identifying the received events. This issue may have been addressed
> already with their
> new version.
>
> This problem should not exist with the event ports interface since a
> user data pointer will
> be returned with the event. The application can differentiate the
> events based on that.
>
> - To de-register or to re-register, the object(file) needs to be
> passed again. But the
> file name could disappear(get removed) from the directory. To
> accommodate this,
> we could change the passed in object to a structure which will
> include the filename.
> Once it gets registered, the system can return an unique id. So
> subsequent actions
> (de-register, re-register), we depend on this 'id'.
>
> Ex.
> file_event {
> uintptr_t id; /* id returned after registering */
> int len; /* length of the file name */
> char fname[0]; /* filename */
> }fobj;
>
>
> So, now the interface will be
>
> port_associate(port, PORT_SOURCE_FILE, (uintptr_t)&fobj, ... )
>
> So this 'fobj' structure can be passed to de-register(port_dissociate)
> or re-register(port_associate again) the file events.
>
>
> FILE_CLOSE_WRITE - What is the purpose of this event? If it is
> found to be useful, we could include it.
>
> Rgds,
> -Prakash.
>>
>>
>> Glynn
>>
>>
>> Glynn
>>
>>>>> Does this mean we don't get told /what/ got created? Is an
>>>>> application that
>>>>> wants to know "what files are disappearing/appearing under
>>>>> /foo/bar/?" going
>>>>> to have to readdir() the whole directory every time it gets an event?
>>>> Otherwise we get into the queued event problem; what happens if the
>>>> application is watching a directory w/ a million files, and someone
>>>> does
>>>> rm * in that directory? Do we generate a million events? Clearly
>>>> there
>>>> are limits to the number of events we can queue in the kernel, esp.
>>>> since the application isn't obligated to read them in a timely
>>>> fashion.
>>>
>>> Forcing a (recursive!) readdir() every time can't scale either; it
>>> just pushes
>>> the problem out all the userspace apps. Perhaps a compromise
>>> approach would
>>> work, so at least the readdir() cost is amortized; i.e. give names
>>> up to a
>>> particular limit.
>>>
>>> Or how do you expect Beagle to be able to work nicely? Is this just
>>> going to
>>> remain something explicitly unsupportable?
>>>
>>>> I'd rather have a model like signals; multiple file writes are
>>>> combined into one event until that event is read by the
>>>> application; any subsequent writes generate another event.
>>>
>>> Would work fine for modifications, yes.
>>>
>>>> I see this as very useful to avoid the {sleep(); stat() } loops we
>>>> often
>>>> see. It's not a mechanism to insert an application as an synchronous
>>>> interposer into the filesystem VOPS.
>>>
>>> I wasn't trying to suggest it was. Synchronisation is neither needed
>>> nor
>>> wanted.
>>>
>>>> The nscd could use this to watch for modifications to configuration
>>>> files rather than stat'ing them before each cache lookup.
>>>
>>> I wasn't suggesting that a non-recursive approach doesn't solve a
>>> whole class
>>> of such situations; it does. In fact, I was merely trying to raise
>>> awareness of
>>> what applications like Beagle actually need in terms of
>>> notifications. If it's
>>> really too hard to do, that's a pity.
>>>
>>> regards,
>>> john
>>> _______________________________________________
>>> perf-discuss mailing list
>>> perf-discuss at opensolaris dot org
>> _______________________________________________
>> perf-discuss mailing list
>> perf-discuss at opensolaris dot org
>
> _______________________________________________
> perf-discuss mailing list
> perf-discuss at opensolaris dot org
_______________________________________________
perf-discuss mailing list
perf-discuss at opensolaris dot org
|
|
|
|
|
Posts:
45
From:
Registered:
3/9/05
|
|
|
|
Re: File events notification mechansim
Posted:
May 5, 2006 1:28 PM
in response to: dkenny
|
|
Darren Kenny wrote:
> Hi Prakash,
>
> I don't think it's the implementation that bridges between the kernel
> and the user spaces that's important to JDS, and probably most other
> people - it's the ultimate API that people will have to write to, and
> I think from this perspective that sysevents is not what we want - it
> may be part of the implementation of the final API, but it shouldn't
> really be the main API that developers have to write to. At the moment
> there are two projects, both with the same ABI (AFAIK), that are
> prevalent in the Linux developer community:
>
I understand. It is the API. We should be implementing to support some
standard API for
file events notification. Unfortunately there is no standard API nor
what the requirements are.
Hence the discussion is around understanding what requirements are for
such a feature.
> * libfam - http://oss.sgi.com/projects/fam/
> * gamin - http://www.gnome.org/~veillard/gamin/index.html
>
> They are compared (with some bias of course) at:
> http://www.gnome.org/~veillard/gamin/differences.html
>
> It's an API like these that we need, so I think any solution proposed
> (albeit sysevents underneath) should expose one of these APIs to the
> user, if the same people produce this API as those that expose things
> via sysevents, then things should work at their optimal. Whether it's
> the SAME API is up for debate, but what ever we expose should at least
> provide the same functionality.
>
When you say 'sysevents' what are you referring to?
Support for a commonly used API, can be provided means of a library.
The implementation will provide a native set of interfaces to support
the desired
functionality/requirements, like using the Event ports interfaces. The
library can
use these native interfaces and expose a common API.
> One think that people have mentioned before as well is the handling of
> distributed file-systems - how do you propose we handle these - this
> is especially important on Solaris given that the majority of our
> customers use NFS for their home directories. From what I understand
> of the File Event Mechanism (FEM) in the kernel, we are using this in
> NFSv4, can this provide us with the ability to see changes to files
> that occur on a NFSv4 filesystem mounted on my desktop, for instance?
>
The proposed solution will not provide file events generated on a
distributed filesystem
from a remote node. But it certainly can provide file events generated
locally on this distributed
filesystem(Ex NFS client side). I don't think the 'FEM' framework in the
kernel, that is
used for the NFSv4 delegation can provide support for events from
remote nodes.
Clearly, this should be transparent to the API.
It appears that the distributed filesystem implementation should
provide necessary means
to collect such events. I think the responsibility falls squarely on the
distributed file system
implementation. I don't of if there are any distributed file system
implementation which can
do that.
-Prakash.
> Thanks,
>
> Darren.
>
>
> Prakash Sangappa wrote:
>> Glynn Foster wrote:
>>>
>>> Yeah, currently Beagle only indexes a relatively small amount of
>>> per-user data, generally in $HOME - however, as has been mentioned,
>>> it's probably one of the first proper use cases of inotify.
>>>
>>> I'd suggest that it's definitely worth looking at what inotify does
>>> - given that there seems to have been a lot of churn with
>>> dnotify/inotify/FAM/gamin/ etc.., there's probably some
>>> implementation lessons to be learned from our Linux neighbours [some
>>> of http://kerneltrap.org/node/3847 might be interesting reading].
>>>
>>> FILE_CLOSE_WRITE is missing from the list of events posted
>>> previously that is apparently useful, as suggested by some GNOME
>>> developers.
>>>
>> Yes, I have looked at some of the issue with 'dnotify' that got
>> addressed
>> by 'inotify' and also some of the issue with inotify.
>>
>> Here is another pointer which lists some issues with 'inotify'. These
>> may have
>> been addressed already with the later version of inotify.
>> http://manic.desrt.ca/inotify
>>
>>
>> Proposed interfaces:
>>
>> - Unlike the 'inotify' interfaces, which uses 'ioctls' to a device,
>> our
>> interfaces will be based on the existing Event ports framework.
>> The events
>> will be delivered to a specified 'Event port'(which is an fd).
>>
>> The Event port can receive events from multiple sources. Currently
>> available event sources in Solaris 'poll, aio, timer, user,
>> message queue(this was recently integrated)'.
>>
>> More event sources can be added.
>>
>> Ex,
>> if the application needs to wait on a 'poll' event and also
>> receive file events
>> notifications, it can receive both these types of events on one
>> Event port.
>>
>> - The application does not have to open the file/directory being
>> watched.
>> Therefore it will not prevent the file from being deleted or
>> unmounting the filesystem. These where issues with 'dnotify'.
>>
>> - If the file system gets unmounted, it will automatically de-register
>> the event notification and send an event indicating that.
>> It can also send a 'file deleted event' when watching a file which
>> gets deleted.
>>
>> - A pointer to some user data can be passed in when registering file
>> event notication.
>> The user data will be returned along with the event notification.
>> This was something
>> missing with the inotify interface.
>>
>> I think 'inotify' returns something called 'watch descriptors' when
>> registering the watches.
>> They had some issue with 'watch descriptors' being reused that would
>> cause confusion in
>> identifying the received events. This issue may have been
>> addressed already with their
>> new version.
>>
>> This problem should not exist with the event ports interface since a
>> user data pointer will
>> be returned with the event. The application can differentiate the
>> events based on that.
>>
>> - To de-register or to re-register, the object(file) needs to be
>> passed again. But the
>> file name could disappear(get removed) from the directory. To
>> accommodate this,
>> we could change the passed in object to a structure which will
>> include the filename.
>> Once it gets registered, the system can return an unique id. So
>> subsequent actions
>> (de-register, re-register), we depend on this 'id'.
>>
>> Ex.
>> file_event {
>> uintptr_t id; /* id returned after
>> registering */
>> int len; /* length of the file name */
>> char fname[0]; /* filename */
>> }fobj;
>>
>>
>> So, now the interface will be
>>
>> port_associate(port, PORT_SOURCE_FILE, (uintptr_t)&fobj, ... )
>>
>> So this 'fobj' structure can be passed to de-register(port_dissociate)
>> or re-register(port_associate again) the file events.
>>
>>
>> FILE_CLOSE_WRITE - What is the purpose of this event? If it is
>> found to be useful, we could include it.
>>
>> Rgds,
>> -Prakash.
>>>
>>>
>>> Glynn
>>>
>>>
>>> Glynn
>>>
>>>>>> Does this mean we don't get told /what/ got created? Is an
>>>>>> application that
>>>>>> wants to know "what files are disappearing/appearing under
>>>>>> /foo/bar/?" going
>>>>>> to have to readdir() the whole directory every time it gets an
>>>>>> event?
>>>>> Otherwise we get into the queued event problem; what happens if the
>>>>> application is watching a directory w/ a million files, and
>>>>> someone does
>>>>> rm * in that directory? Do we generate a million events? Clearly
>>>>> there
>>>>> are limits to the number of events we can queue in the kernel, esp.
>>>>> since the application isn't obligated to read them in a timely
>>>>> fashion.
>>>>
>>>> Forcing a (recursive!) readdir() every time can't scale either; it
>>>> just pushes
>>>> the problem out all the userspace apps. Perhaps a compromise
>>>> approach would
>>>> work, so at least the readdir() cost is amortized; i.e. give names
>>>> up to a
>>>> particular limit.
>>>>
>>>> Or how do you expect Beagle to be able to work nicely? Is this just
>>>> going to
>>>> remain something explicitly unsupportable?
>>>>
>>>>> I'd rather have a model like signals; multiple file writes are
>>>>> combined into one event until that event is read by the
>>>>> application; any subsequent writes generate another event.
>>>>
>>>> Would work fine for modifications, yes.
>>>>
>>>>> I see this as very useful to avoid the {sleep(); stat() } loops we
>>>>> often
>>>>> see. It's not a mechanism to insert an application as an synchronous
>>>>> interposer into the filesystem VOPS.
>>>>
>>>> I wasn't trying to suggest it was. Synchronisation is neither
>>>> needed nor
>>>> wanted.
>>>>
>>>>> The nscd could use this to watch for modifications to configuration
>>>>> files rather than stat'ing them before each cache lookup.
>>>>
>>>> I wasn't suggesting that a non-recursive approach doesn't solve a
>>>> whole class
>>>> of such situations; it does. In fact, I was merely trying to raise
>>>> awareness of
>>>> what applications like Beagle actually need in terms of
>>>> notifications. If it's
>>>> really too hard to do, that's a pity.
>>>>
>>>> regards,
>>>> john
>>>> _______________________________________________
>>>> perf-discuss mailing list
>>>> perf-discuss at opensolaris dot org
>>> _______________________________________________
>>> perf-discuss mailing list
>>> perf-discuss at opensolaris dot org
>>
>> _______________________________________________
>> perf-discuss mailing list
>> perf-discuss at opensolaris dot org
> _______________________________________________
> perf-discuss mailing list
> perf-discuss at opensolaris dot org
_______________________________________________
perf-discuss mailing list
perf-discuss at opensolaris dot org
|
|
|
|
|
Posts:
213
From:
US
Registered:
3/9/05
|
|
|
|
Re: File events notification mechansim
Posted:
May 5, 2006 1:48 PM
in response to: praks
|
|
On Fri, Prakash Sangappa wrote:
> Darren Kenny wrote:
> >Hi Prakash,
> >
> >I don't think it's the implementation that bridges between the kernel
> >and the user spaces that's important to JDS, and probably most other
> >people - it's the ultimate API that people will have to write to, and
> >I think from this perspective that sysevents is not what we want - it
> >may be part of the implementation of the final API, but it shouldn't
> >really be the main API that developers have to write to. At the moment
> >there are two projects, both with the same ABI (AFAIK), that are
> >prevalent in the Linux developer community:
> >
> I understand. It is the API. We should be implementing to support some
> standard API for
> file events notification. Unfortunately there is no standard API nor
> what the requirements are.
> Hence the discussion is around understanding what requirements are for
> such a feature.
>
>
> > * libfam - http://oss.sgi.com/projects/fam/
> > * gamin - http://www.gnome.org/~veillard/gamin/index.html
> >
> >They are compared (with some bias of course) at:
> >http://www.gnome.org/~veillard/gamin/differences.html
> >
> >It's an API like these that we need, so I think any solution proposed
> >(albeit sysevents underneath) should expose one of these APIs to the
> >user, if the same people produce this API as those that expose things
> >via sysevents, then things should work at their optimal. Whether it's
> >the SAME API is up for debate, but what ever we expose should at least
> >provide the same functionality.
> >
> When you say 'sysevents' what are you referring to?
>
> Support for a commonly used API, can be provided means of a library.
> The implementation will provide a native set of interfaces to support
> the desired
> functionality/requirements, like using the Event ports interfaces. The
> library can
> use these native interfaces and expose a common API.
>
> >One think that people have mentioned before as well is the handling of
> >distributed file-systems - how do you propose we handle these - this
> >is especially important on Solaris given that the majority of our
> >customers use NFS for their home directories. From what I understand
> >of the File Event Mechanism (FEM) in the kernel, we are using this in
> >NFSv4, can this provide us with the ability to see changes to files
> >that occur on a NFSv4 filesystem mounted on my desktop, for instance?
> >
> The proposed solution will not provide file events generated on a
> distributed filesystem
> from a remote node. But it certainly can provide file events generated
> locally on this distributed
> filesystem(Ex NFS client side). I don't think the 'FEM' framework in the
> kernel, that is
> used for the NFSv4 delegation can provide support for events from
> remote nodes.
> Clearly, this should be transparent to the API.
>
> It appears that the distributed filesystem implementation should
> provide necessary means
> to collect such events. I think the responsibility falls squarely on the
> distributed file system
> implementation. I don't of if there are any distributed file system
> implementation which can
> do that.
Did my email about the upcoming NFSv4.1 directory delegation feature
not make it to the alias? No, I checked, it is there. :-)
NFSv4.1 has a notification mechanism and it has its own set of
restrictions but NFSv4.1 will allow for a certain set of notifications
to be sent to a client that requests them.
You should also note that CIFS also has file notfication methods as well.
Spencer
_______________________________________________
perf-discuss mailing list
perf-discuss at opensolaris dot org
|
|
|
|
|
Posts:
45
From:
Registered:
3/9/05
|
|
|
|
Re: File events notification mechansim
Posted:
May 5, 2006 3:18 PM
in response to: shepler
|
|
Spencer Shepler wrote:
>> It appears that the distributed filesystem implementation should
>> provide necessary means
>> to collect such events. I think the responsibility falls squarely on the
>> distributed file system
>> implementation. I don't of if there are any distributed file system
>> implementation which can
>> do that.
>>
>
> Did my email about the upcoming NFSv4.1 directory delegation feature
> not make it to the alias? No, I checked, it is there. :-)
>
Sorry I missed that.
> NFSv4.1 has a notification mechanism and it has its own set of
> restrictions but NFSv4.1 will allow for a certain set of notifications
> to be sent to a client that requests them.
>
> You should also note that CIFS also has file notfication methods as well.
>
>
Great!. That will be certainly useful. Are these notification types
proposed for
NFSv4.1 similar to what CIFS provides?
What are these notification types that the client can ask?
-Prakash.
> Spencer
>
>
> _______________________________________________
> perf-discuss mailing list
> perf-discuss at opensolaris dot org
>
_______________________________________________
perf-discuss mailing list
perf-discuss at opensolaris dot org
|
|
|
|
|
Posts:
213
From:
US
Registered:
3/9/05
|
|
|
|
Re: File events notification mechansim
Posted:
May 8, 2006 7:55 AM
in response to: praks
|
|
On Fri, Prakash Sangappa wrote:
> Spencer Shepler wrote:
> >>It appears that the distributed filesystem implementation should
> >>provide necessary means
> >>to collect such events. I think the responsibility falls squarely on the
> >>distributed file system
> >>implementation. I don't of if there are any distributed file system
> >>implementation which can
> >>do that.
> >>
> >
> >Did my email about the upcoming NFSv4.1 directory delegation feature
> >not make it to the alias? No, I checked, it is there. :-)
> >
> Sorry I missed that.
> >NFSv4.1 has a notification mechanism and it has its own set of
> >restrictions but NFSv4.1 will allow for a certain set of notifications
> >to be sent to a client that requests them.
> >
> >You should also note that CIFS also has file notfication methods as well.
> >
> >
> Great!. That will be certainly useful. Are these notification types
> proposed for
> NFSv4.1 similar to what CIFS provides?
I don't know and am not aware of a quick-guide for comparison .
>
> What are these notification types that the client can ask?
For NFSv4.1 the client can ask for remove,add,rename actions
within a directory and can also ask for notification of attribute
changes for the directory object itself and the objects within
the directory.
Spencer
_______________________________________________
perf-discuss mailing list
perf-discuss at opensolaris dot org
|
|
|
|
|
Michael Pogue
Michael.Pogue@Sun.COM
|
|
|
|
Re: File events notification mechansim
Posted:
May 4, 2006 1:32 PM
in response to: johnlev
|
|
See also Apple's Spotlight, which is similar to Linux's Beagle. I believe that
Windows-Next will also have this capability. So, yes, I think this should be a
primary use case for this new API.
Mike
John Levon wrote:
> On Wed, May 03, 2006 at 07:31:08PM -0700, Prakash Sangappa wrote:
>
>
>>If you where to watch for events on an entire directory tree, what types
>>of events that would be?
>
>
> Presumably there would be some way to specify, but file creation/deletion would
> be the most obviously useful events.
>
>
>>How would this be useful?
>
>
> Beagle wants it, I think:
>
> http://beaglewiki.org/Main_Page
>
> as well as some other GNOME things, apparently.
>
>
>>can be to just a watch on the directory "/etc". The FILE_CREATE event would
>>mean that a file or a directory was created under "/etc". Once the
>>application gets this event, it can go and check if the file got created.
>
>
> Does this mean we don't get told /what/ got created? Is an application that
> wants to know "what files are disappearing/appearing under /foo/bar/?" going to
> have to readdir() the whole directory every time it gets an event?
>
>
>>>How does it interact with unmounting the underlying filesystem?
>>
>>It will hold a reference to the 'vnode' of the file/directory in the kernel,
>>while we have the file events monitor on it. If the file system gets
>>unmounted, it will de-register the monitor and send an exception event saying
>>the filesystem got unmounted. These exception events need to be defined.
>
>
> Sounds sensible...
>
> regards
> john
> _______________________________________________
> perf-discuss mailing list
> perf-discuss at opensolaris dot org
_______________________________________________
perf-discuss mailing list
perf-discuss at opensolaris dot org
|
|
|
|
|
Posts:
213
From:
US
Registered:
3/9/05
|
|
|
|
Re: File events notification mechansim
Posted:
May 4, 2006 1:50 PM
in response to: Michael Pogue
|
|
On Thu, Michael Pogue wrote:
> See also Apple's Spotlight, which is similar to Linux's Beagle. I believe
> that Windows-Next will also have this capability. So, yes, I think this
> should be a primary use case for this new API.
>
> Mike
>
> John Levon wrote:
> >On Wed, May 03, 2006 at 07:31:08PM -0700, Prakash Sangappa wrote:
> >
> >
> >>If you where to watch for events on an entire directory tree, what types
> >>of events that would be?
> >
> >
> >Presumably there would be some way to specify, but file creation/deletion
> >would
> >be the most obviously useful events.
I would like to point out that the proposal for NFSv4's first minor
version contains a directory delegation feature. Within that is a
notification mechanism. The client is able to ask for asynchronous
notifications on a directory (and the objects within it) for
various events. Those events include things like:
adding a file
removing a file
renameing a file
attribute changes (for directory and objects within it)
cookie verifier change (really an NFSv4 thing)
The scope of a notification is just the directory and objects contained
within it (not further down the tree).
As mentioned, the events are asynchronous in the sense that the server
will not delay operations just because it needs to notify a client
of one of the above events. The client gets to ask for time granularity
of the notifications. The server is not obliged to provide
notifications (not supported or too many requests, etc.).
Thought this would be helpful information for the discussion.
Spencer
_______________________________________________
perf-discuss mailing list
perf-discuss at opensolaris dot org
|
|
|
|
|
Eoin Lawless
Eoin.Lawless@Sun.COM
|
|
|
|
Re: File events notification mechansim
Posted:
May 8, 2006 5:53 AM
in response to: Michael Pogue
|
|
As far as I know, Beagle is similar to Spotlight rather than the other
way round - it tries to copy Spotlight. It was very far behind Spotlight
when I last tried it.
A good article on file system events, indexing and search in Apple's
MacosX 10.4 Tiger is at:
http://arstechnica.com/reviews/os/macosx-10.4.ars/
(sections 6 to 12).
Eoin
On Thu, 2006-05-04 at 21:32, Michael Pogue wrote:
> See also Apple's Spotlight, which is similar to Linux's Beagle. I believe that
> Windows-Next will also have this capability. So, yes, I think this should be a
> primary use case for this new API.
>
> Mike
>
> John Levon wrote:
> > On Wed, May 03, 2006 at 07:31:08PM -0700, Prakash Sangappa wrote:
> >
> >
> >>If you where to watch for events on an entire directory tree, what types
> >>of events that would be?
> >
> >
> > Presumably there would be some way to specify, but file creation/deletion would
> > be the most obviously useful events.
> >
> >
> >>How would this be useful?
> >
> >
> > Beagle wants it, I think:
> >
> > http://beaglewiki.org/Main_Page
> >
> > as well as some other GNOME things, apparently.
> >
> >
> >>can be to just a watch on the directory "/etc". The FILE_CREATE event would
> >>mean that a file or a directory was created under "/etc". Once the
> >>application gets this event, it can go and check if the file got created.
> >
> >
> > Does this mean we don't get told /what/ got created? Is an application that
> > wants to know "what files are disappearing/appearing under /foo/bar/?" going to
> > have to readdir() the whole directory every time it gets an event?
> >
> >
> >>>How does it interact with unmounting the underlying filesystem?
> >>
> >>It will hold a reference to the 'vnode' of the file/directory in the kernel,
> >>while we have the file events monitor on it. If the file system gets
> >>unmounted, it will de-register the monitor and send an exception event saying
> >>the filesystem got unmounted. These exception events need to be defined.
> >
> >
> > Sounds sensible...
> >
> > regards
> > john
> > _______________________________________________
> > perf-discuss mailing list
> > perf-discuss at opensolaris dot org
> _______________________________________________
> perf-discuss mailing list
> perf-discuss at opensolaris dot org
_______________________________________________
perf-discuss mailing list
perf-discuss at opensolaris dot org
|
|
|
|
|
Posts:
45
From:
Registered:
3/9/05
|
|
|
|
Re: File events notification mechansim
Posted:
May 10, 2006 3:28 PM
in response to: Eoin Lawless
|
|
I found the following which talks about the file event notification
facility in Apples OS X 10.4. The fslogger is a utility which uses this
feature. Apparently this is the same feature used by 'Spotlight' too
http://www.kernelthread.com/software/fslogger
Their file events notification mechanism is available only to root. It
also mentions that the events get queued up and once the queue is full,
the kernel will drop events. But it will send an event indicating that
the queue is full. I Linux's 'inotify' implmenentation is also similar.
It sends a 'OVERFLOW' event when the queue gets full.
Spotlight..
Is a system service for searching files in a desktop environment.
>From reading the article that Eoin has provided, tt can accept complex
search queries based on file attributes. It uses extended file
attributes(called metadata)for indexing. It has APIs defined so that
applications can use to interact with this service.
It appears to use file events to monitor for any changes to files
and directories on all the filesystems on the desktop and update its
index data base. Obviously this would not be very effective on
any distributed filesystem mounted.
If it is possible for Spotlight to miss events, since the
kernel can drop it when the queue is full, it must use
some user land search mechanism as a backup. I could not
see this mentioned in the article.
So, if that is the requirement, we can have similar functionality
on Solaris too. The queue size could be specified when registering
to watch the events. There should be a hard limit defined
so that the system does not get overwhelmed by the flood
of events. An out of band OVERFLOW event can be sent once the queue
gets full. The application will have to take appropriate action if
the queue get full and it receives an OVERFLOW event.
We could restrict watching file events, on an entire filesystem/directory
tree or on all the filesystems in the system, only to root. But allow
watching an individual file or directory to all users.
If the purpose of watching file events on an entire filesystem
is for indexing like what Spotlight does, then this should
be provided as system service and have API's for applications to
interact with this service, rather then different applications trying
to duplicate it.
-Prakash.
Eoin Lawless wrote:
> As far as I know, Beagle is similar to Spotlight rather than the other
> way round - it tries to copy Spotlight. It was very far behind Spotlight
> when I last tried it.
>
> A good article on file system events, indexing and search in Apple's
> MacosX 10.4 Tiger is at:
> http://arstechnica.com/reviews/os/macosx-10.4.ars/
> (sections 6 to 12).
>
> Eoin
>
>
> On Thu, 2006-05-04 at 21:32, Michael Pogue wrote:
>
>> See also Apple's Spotlight, which is similar to Linux's Beagle. I believe that
>> Windows-Next will also have this capability. So, yes, I think this should be a
>> primary use case for this new API.
>>
>> Mike
>>
>> John Levon wrote:
>>
>>> On Wed, May 03, 2006 at 07:31:08PM -0700, Prakash Sangappa wrote:
>>>
>>>
>>>
>>>> If you where to watch for events on an entire directory tree, what types
>>>> of events that would be?
>>>>
>>> Presumably there would be some way to specify, but file creation/deletion would
>>> be the most obviously useful events.
>>>
>>>
>>>
>>>> How would this be useful?
>>>>
>>> Beagle wants it, I think:
>>>
>>> http://beaglewiki.org/Main_Page
>>>
>>> as well as some other GNOME things, apparently.
>>>
>>>
>>>
>>>> can be to just a watch on the directory "/etc". The FILE_CREATE event would
>>>> mean that a file or a directory was created under "/etc". Once the
>>>> application gets this event, it can go and check if the file got created.
>>>>
>>> Does this mean we don't get told /what/ got created? Is an application that
>>> wants to know "what files are disappearing/appearing under /foo/bar/?" going to
>>> have to readdir() the whole directory every time it gets an event?
>>>
>>>
>>>
>>>>> How does it interact with unmounting the underlying filesystem?
>>>>>
>>>> It will hold a reference to the 'vnode' of the file/directory in the kernel,
>>>> while we have the file events monitor on it. If the file system gets
>>>> unmounted, it will de-register the monitor and send an exception event saying
>>>> the filesystem got unmounted. These exception events need to be defined.
>>>>
>>> Sounds sensible...
>>>
>>> regards
>>> john
>>> _______________________________________________
>>> perf-discuss mailing list
>>> perf-discuss at opensolaris dot org
>>>
>> _______________________________________________
>> perf-discuss mailing list
>> perf-discuss at opensolaris dot org
>>
>
>
_______________________________________________
perf-discuss mailing list
perf-discuss at opensolaris dot org
|
|
|
|
|
Eoin Lawless
Eoin.Lawless@Sun.COM
|
|
|
|
Re: File events notification mechansim
Posted:
May 11, 2006 3:14 AM
in response to: praks
|
|
On Wed, 2006-05-10 at 23:28, prakash sangappa wrote:
Spotlight.
The distributed filesystem issue is somewhat addressed by remote
spotlight:
http://weblog.infoworld.com/enterprisemac/archives/2006/04/remote_desktop.html
"Remote Spotlight is a hugely useful feature. It launches Spotlight
searches
on multiple remote clients simultaneously and returns the results fast,
grouped
by client, to the ARD console. In other words, it's a LAN-wide Spotlight
search
with the same criteria you use with Tiger Spotlight, and uses each
machine's
local index and application-specific importers. Like Tiger Spotlight,
you can
narrow the results with qualifiers after the results are displayed."
A full index of a filesystem can be started manually - in fact this
happens
the first time OS X 10.4 is booted up.
I was very impressed when I first used Spotlight. Start a spotlight
search for 'foobar'. In Terminal do "vi file.txt" type in 'foobar',
save and file.txt instantly appears in the spotlight search window.
However, if you call the file just 'file' it is missed! - a side effect
of the way the indexing/importing works. Nevertheless it's far and away
the best _desktop_ search system I've yet used.
> I found the following which talks about the file event notification
> facility in Apples OS X 10.4. The fslogger is a utility which uses this
> feature. Apparently this is the same feature used by 'Spotlight' too
>
> http://www.kernelthread.com/software/fslogger
>
> Their file events notification mechanism is available only to root. It
> also mentions that the events get queued up and once the queue is full,
> the kernel will drop events. But it will send an event indicating that
> the queue is full. I Linux's 'inotify' implmenentation is also similar.
> It sends a 'OVERFLOW' event when the queue gets full.
>
>
> Spotlight..
> Is a system service for searching files in a desktop environment.
> From reading the article that Eoin has provided, tt can accept complex
> search queries based on file attributes. It uses extended file
> attributes(called metadata)for indexing. It has APIs defined so that
> applications can use to interact with this service.
>
> It appears to use file events to monitor for any changes to files
> and directories on all the filesystems on the desktop and update its
> index data base. Obviously this would not be very effective on
> any distributed filesystem mounted.
>
> If it is possible for Spotlight to miss events, since the
> kernel can drop it when the queue is full, it must use
> some user land search mechanism as a backup. I could not
> see this mentioned in the article.
>
>
> So, if that is the requirement, we can have similar functionality
> on Solaris too. The queue size could be specified when registering
> to watch the events. There should be a hard limit defined
> so that the system does not get overwhelmed by the flood
> of events. An out of band OVERFLOW event can be sent once the queue
> gets full. The application will have to take appropriate action if
> the queue get full and it receives an OVERFLOW event.
>
> We could restrict watching file events, on an entire filesystem/directory
> tree or on all the filesystems in the system, only to root. But allow
> watching an individual file or directory to all users.
>
> If the purpose of watching file events on an entire filesystem
> is for indexing like what Spotlight does, then this should
> be provided as system service and have API's for applications to
> interact with this service, rather then different applications trying
> to duplicate it.
>
> -Prakash.
>
> Eoin Lawless wrote:
> > As far as I know, Beagle is similar to Spotlight rather than the other
> > way round - it tries to copy Spotlight. It was very far behind Spotlight
> > when I last tried it.
> >
> > A good article on file system events, indexing and search in Apple's
> > MacosX 10.4 Tiger is at:
> > http://arstechnica.com/reviews/os/macosx-10.4.ars/
> > (sections 6 to 12).
> >
> > Eoin
> >
> >
> > On Thu, 2006-05-04 at 21:32, Michael Pogue wrote:
> >
> >> See also Apple's Spotlight, which is similar to Linux's Beagle. I believe that
> >> Windows-Next will also have this capability. So, yes, I think this should be a
> >> primary use case for this new API.
> >>
> >> Mike
> >>
> >> John Levon wrote:
> >>
> >>> On Wed, May 03, 2006 at 07:31:08PM -0700, Prakash Sangappa wrote:
> >>>
> >>>
> >>>
> >>>> If you where to watch for events on an entire directory tree, what types
> >>>> of events that would be?
> >>>>
> >>> Presumably there would be some way to specify, but file creation/deletion would
> >>> be the most obviously useful events.
> >>>
> >>>
> >>>
> >>>> How would this be useful?
> >>>>
> >>> Beagle wants it, I think:
> >>>
> >>> http://beaglewiki.org/Main_Page
> >>>
> >>> as well as some other GNOME things, apparently.
> >>>
> >>>
> >>>
> >>>> can be to just a watch on the directory "/etc". The FILE_CREATE event would
> >>>> mean that a file or a directory was created under "/etc". Once the
> >>>> application gets this event, it can go and check if the file got created.
> >>>>
> >>> Does this mean we don't get told /what/ got created? Is an application that
> >>> wants to know "what files are disappearing/appearing under /foo/bar/?" going to
> >>> have to readdir() the whole directory every time it gets an event?
> >>>
> >>>
> >>>
> >>>>> How does it interact with unmounting the underlying filesystem?
> >>>>>
> >>>> It will hold a reference to the 'vnode' of the file/directory in the kernel,
> >>>> while we have the file events monitor on it. If the file system gets
> >>>> unmounted, it will de-register the monitor and send an exception event saying
> >>>> the filesystem got unmounted. These exception events need to be defined.
> >>>>
> >>> Sounds sensible...
> >>>
> >>> regards
> >>> john
> >>> _______________________________________________
> >>> perf-discuss mailing list
> >>> perf-discuss at opensolaris dot org
> >>>
> >> _______________________________________________
> >> perf-discuss mailing list
> >> perf-discuss at opensolaris dot org
> >>
> >
> >
_______________________________________________
perf-discuss mailing list
perf-discuss at opensolaris dot org
|
|
|
|
|
Posts:
45
From:
Registered:
3/9/05
|
|
|
|
Re: File events notification mechansim
Posted:
May 11, 2006 1:31 PM
in response to: praks
|
|
[snip]
We have a good discussion going. To summarize...
File event types
FILE_ACCESS File was accessed
FILE_MODIFIED File was modified.
FILE_CREATE File/directory was created.
FILE_DELETE File/directory was deleted.
FILE_ATTRIB File/directory attribute changed
FILE_OPEN File was opened.
FILE_CLOSE_WRITE File that was modified got closed.
FILE_CLOSE_NOWRITE File was closed without modification.
FILE_RENAME_FROM File was renamed from this name.
FILE_RENAME_TO File was renamed to this name.
Exception events:
UNMOUNTED Watched filesystem got umounted.
DELETED File that was being watched got deleted.
OVERFLOW Events queue is full.
Additional directives,
NOTIFY_ONCE De-register after the first event delivery.
Proposal was to report multiple events as on event, like signals, till it
is read(no queuing). Also, the file events can be watched on a file or a
directory.
The interfaces will be based on the event ports interfaces. If there is
a commonly
used API, like the Linux's 'inotify' we could provide that using a library.
There appears to be a need for queuing events. Each event can provide
additional
details other then the event, ex file name when a file is created. We
could possible
provide this with some limit set on the queue size. This will address
issues with
events being generated at a faster rate then they can be read, call it
queuing issue.
Once the number of events queued exceed the limit, an OVERFLOW event is
sent to indicate
that. The subsequent events will be dropped by the system. Once the
application
receives an OVERFLOW event it can perform necessary action to find out
what has changed.
For use cases like 'Spotlight & Beagle', we need provision to watch for
file events
occurring on an entire system i.e all the mounted file systems. This can
be a scalability
issue and it also raises security concerns. These issues could be
addressed to some
extent if watching file events on an entire filesystem can be restricted
to the
'root' user. The queuing issues mentioned above will apply. Also the
event types
that can be supported when watching an entire filesystem can be a subset
of the
event types mentioned above, Ex FILE_CREATE, FILE_MODIFIED, FILE_DELETE
We can allow non root user applications to watch for file events on an
individual
file or directory.
The Spotlight like service for indexing file metadata and enabling
searches, will
have to run as root and do the necessary security checks before it can
share the
search results to non root users. This type of application will have to
have backup
procedure to collect changes that occur on the filesystem, in the case
of an OVERFLOW
event, as file events can be dropped by the kernel.
The file events from a remote node on a distributed filesystem will be
delivered as
long as the distributed filesystem has support to do so. Ex. NFSv4.1
implemention will
provide support for some types of file events. The CIFS filesystem
already has support
for this. There may not be a standard around the remote node event
notification. This
needs further thought.
The remote Spotlight implementation seems to address remote node file
change events issue
at the application level. This service will directly interact with the
remote node
Spotlight service to do the search. Not sure how are they planning on
supporting this in
an heterogeneous environment.
Behavior :
The implementation will be added as a new event source to event ports
interface.
The events will be delivered to an event port, that need to be passed
when registering.
The event port is an 'fd' and this fd is pollable.
After registering to watch events on a file or a directory, the default
behavior will be to continue to deliver events until it is de-registered.
If the 'NOTIFY_ONCE' directive is specified, it will deliver one event and
de-register automatically. This file has to be registered again to receive
file events.
The event types, being watched, can be changed by re-registering the file.
The new events are passed while re-registering.
After a file has been unlinked(removed), it will continue to watch
for events as long as this file has open references. It will deliver a
'DELETED' event when the last open references to this unlinked file goes
away.
If the filesystem on which the files are being watched is unmounted it will
de-register the watch and send an 'UNMOUNTED' event indicating that the
filesystem
has been unmounted.
These points are mainly concerning the requirements which will help
shape the API.
Thanks,
-Prakash.
_______________________________________________
perf-discuss mailing list
perf-discuss at opensolaris dot org
|
|
|
|
|
Posts:
12
From:
Germany
Registered:
10/14/05
|
|
|
|
Re: File events notification mechansim
Posted:
May 13, 2006 3:34 AM
in response to: praks
|
|
Hey Prakash,
2 comments see below:
prakash sangappa wrote:
PS> [snip]
PS>
PS> We have a good discussion going. To summarize...
PS>
PS>
PS> File event types
PS>
PS> FILE_ACCESS File was accessed
PS> FILE_MODIFIED File was modified.
PS> FILE_CREATE File/directory was created.
PS> FILE_DELETE File/directory was deleted.
PS> FILE_ATTRIB File/directory attribute changed
PS> FILE_OPEN File was opened.
PS> FILE_CLOSE_WRITE File that was modified got closed.
PS> FILE_CLOSE_NOWRITE File was closed without modification.
PS> FILE_RENAME_FROM File was renamed from this name.
PS> FILE_RENAME_TO File was renamed to this name.
PS>
PS>
PS> Exception events:
PS>
PS> UNMOUNTED Watched filesystem got umounted.
PS> DELETED File that was being watched got deleted.
PS> OVERFLOW Events queue is full.
PS>
PS>
PS> Additional directives,
PS> NOTIFY_ONCE De-register after the first event
PS> delivery.
PS...
PS> Proposal was to report multiple events as on event, like signals,
PS> till it
PS> is read(no queuing). Also, the file events can be watched on a file or a
PS> directory.
To be more precise: multiple events of the same kind as one event
(but clearly to differentiate between different file events delivered
as different "notifications" - or - to distinguish between different
files being e.g. opened as separate events).
So e.g., a change of the same file multiple times would result in generating
an additional "event" each time the file is e.g. changed. If the
"consumer" (aka application) has not "processed" the event - fine .
The "bit" mask for exactly that "event" would be set to "1" each time
the file is changed. ((Needless to say, that event ports guarantees
no race conditions here ... ))
When another file that is "monitored" is changed, it would result in
another event (clearly).
So this is still true:
PS> Each event can provide additional
PS> details other then the event, ex file name when a file is created
If we go for this (which has its nice sides, too, I agree), than
we don't need the above:
PS> There appears to be a need for queuing events. ... We
PS> could possible
PS> provide this with some limit set on the queue size. This will address
PS> issues with
PS> events being generated at a faster rate then they can be read, call it
PS> queuing issue.
PS> Once the number of events queued exceed the limit, an OVERFLOW event is
PS> sent to indicate
PS> that. The subsequent events will be dropped by the system. Once the
PS> application
PS> receives an OVERFLOW event it can perform necessary action to find out
PS> what has changed.
Best regards,
Michael
--
Michael Schulte mschulte at sunspezialist dot de
OpenSolaris Kernel Development http://opensolaris.org/
_______________________________________________
perf-discuss mailing list
perf-discuss at opensolaris dot org
|
|
|
|
|
Posts:
1,580
From:
US
Registered:
6/14/05
|
|
|
|
Re: File events notification mechansim
Posted:
Dec 13, 2006 9:08 PM
in response to: praks
To: Communities » performance » discuss
|
|
[...]
> The interfaces will be based on the event ports
> interfaces. If there is
> a commonly
> used API, like the Linux's 'inotify' we could provide
> that using a library.
[...]
From what I've read, in terms of what they do, there's perhaps room for
two APIs: one like gamin (as a superset of FAM), and one like libevent.
See http://oss.sgi.com/archives/fam/2003-10/msg00014.html
I'm not sure how good coverage those two would provide of existing
open source apps...
|
|
|
|
|
Posts:
45
From:
Registered:
3/9/05
|
|
|
|
Re: Re: File events notification mechansim
Posted:
Dec 14, 2006 6:36 PM
in response to: rlhamil
|
|
Gamin/FAM is different from libevent. Gamin/FAM provides file
monitoring services
where as 'libevent' provides a general event notification API to
collect events from different
sources.
Libevent interfaces with the event notification framework that the OS
provides. The
the event notification framework we have on Solaris is 'Event Ports' .
David Pacheco has
provided a patch for libevent to make it work with event ports.
Coming back to the file events notification, yes, we are looking at
porting Gamin(Which is based on
FAM) to use the file events notification API.
In addition to that we are looking at implementing a library which
provides a set of interfaces that
will help do the necessary filtering and caching of file events
information. These interfaces will interact
with the file events API. An interface, to collect the list of changes
from a filesystem or
a directory tree, like it was discussed previously on this forum, will
be included. They will return
a handle of some sort.
Just as an idea, the interfaces can be as follows(fen - for file events
notification):
To start watching a file, directory or a directrory tree:-
handle_t fen_watch(int port, char *fname, time, ...)
handle_t fen_watch_dirtree(int port, char *path, time, ...)
Once we get event notification that will be collected by the
'port_get(port, ...)' interface,
the following can be called to collect the list of changes. Filtering
can be done
based on the types of change events requested.
int fen_get_dirtree_changes(handle_t hd, time, ....)
int fen_get_dir_changes(handle_t hd, time, ....)
int fen_get_file_changes(handle_t hd, time, ...)
For example if a directory modification event is received, then
fen_get_dir_changes() can be called to collect, say, the list
of files that where created.
Similarly, fen_get_dirtree_changes() can be called to collect
changes under a directory tree.
To cancel a watch:
int fen_cancel(handle_t hd)
The details need to be worked out. The application can use the same
event port to collect events from different sources. It would wait
for events using the port_get() interface.
-Prakash.
Richard L. Hamilton wrote:
> [...]
>
>> The interfaces will be based on the event ports
>> interfaces. If there is
>> a commonly
>> used API, like the Linux's 'inotify' we could provide
>> that using a library.
>>
> [...]
>
> >From what I've read, in terms of what they do, there's perhaps room for
> two APIs: one like gamin (as a superset of FAM), and one like libevent.
> See http://oss.sgi.com/archives/fam/2003-10/msg00014.html
> I'm not sure how good coverage those two would provide of existing
> open source apps...
>
>
> This message posted from opensolaris.org
> _______________________________________________
> perf-discuss mailing list
> perf-discuss at opensolaris dot org
>
_______________________________________________
perf-discuss mailing list
perf-discuss at opensolaris dot org
|
|
|
|
|
Posts:
8
From:
Registered:
4/3/06
|
|
|
|
Re: File events notification mechansim
Posted:
Jun 6, 2006 3:44 AM
in response to: praks
|
|
> Here are a some of the file event types considered.
>
> FILE_OPEN File opened.
> FILE_CLOSE File closed.
> FILE_WRITE File was written to.
> FILE_READ File was read.
> FILE_ATTRIB_CHANGE File attributes changed
> FILE_CREATE File/Directory was created.
> FILE_REMOVE/UNLINK File/Directory was removed/deleted.
> FILE_RENAME_TO/FROM File was renamed to/from.
How about file locking/unlocking?
If a file was resized (truncated), how would that be represented? It might also be an idea to be able to differentiate between file-writing and file-appending. What would happen if one file was renamed over another - would you get one event or two (one for rename, one for old file effectively being deleted)?
As a related concept, does anyone know how easy or practical it would be to use ZFS's copy-on-write update mecanism to be able to read a "log" (effectively) of recent changes to a file-system? I'm thinking of an API where you start by passing a start date/time, and then can iterate through each change in sequence. Obviously, there would be no gurantees on the scope (in terms of date range) of such data.
It's hard to think of good usage cases right away though, and the type of applications/situation that could make use of such a thing would probably be rather different to event-notification applications. But, say your system just rebooted and you wanted to get an idea of what files were changed (and how) just prior to the reboot. Or you want to get an idea of recent (but not current) activity by a particular user, or to a particular directory tree.
|
|
|
|
|
Posts:
12
From:
Germany
Registered:
10/14/05
|
|
|
|
Re: Re: File events notification mechansim
Posted:
Jun 9, 2006 12:15 PM
in response to: c.rijk
|
|
Hey Chris,
see some comments from my point of view inline.
Chris Rijk schrieb:
>> Here are a some of the file event types considered.
>>
>> FILE_OPEN File opened.
>> FILE_CLOSE File closed.
>> FILE_WRITE File was written to.
>> FILE_READ File was read.
>> FILE_ATTRIB_CHANGE File attributes changed
>> FILE_CREATE File/Directory was created.
>> FILE_REMOVE/UNLINK File/Directory was removed/deleted.
>> FILE_RENAME_TO/FROM File was renamed to/from.
>
> How about file locking/unlocking?
Could be realized as FILE_ATTRIB_CHANGE notification.
>
> If a file was resized (truncated), how would that be represented?
Could be realized as FILE_WRITE event.
> It might also be an idea to be able to differentiate between file-writing and file-appending.
might be. But I think it is task of the application programmer
to react to the event FILE_WRITE if he is interested in differentiating
between these facts.
> What would happen if one file was renamed over another - would you get one event
> or two (one for rename, one for old file effectively being deleted)?
This is an interesting question that should be answered automatically
when you take into account that file events are "bound" to particular
files (or a list of them or a directory). And this is the answer. If you
have "subscribed" to get events for the old file, you get FILE_REMOVE;
if you are "observing" the old file and the e.g. directory, you get two.
>
>
> As a related concept, does anyone know how easy or practical it would be to
> use ZFS's copy-on-write update mecanism to be able to read a "log" (effectively)
> of recent changes to a file-system? I'm thinking of an API where you start by passing
> a start date/time, and then can iterate through each change in sequence. Obviously,
> there would be no gurantees on the scope (in terms of date range) of such data.
Shouldn't be a problem as file events use event ports and these are
independent of the underlying filesystem (UFS, ZFS, NFS?).
Best regards,
Michael
_______________________________________________
perf-discuss mailing list
perf-discuss at opensolaris dot org
|
|
|
|
|
Posts:
3
From:
San Diego
Registered:
8/21/08
|
|
|
|
Re: File events notification mechansim
Posted:
Aug 22, 2008 12:06 AM
in response to: praks
To: Communities » performance » discuss
Cc: OpenSolaris » help
|
|
The file event discussed here in May 2006 is close to what I am looking for. Its only shortcoming is that a separate event is needed for each directory being tracked. A more useful interface would allow events to be received for all create/delete activity in a directory tree, starting from a given node. In his blog entry dated 27 Aug 2007, Prakash noted that the functionality (FEN) would be available in the Nevada build 72. As of 20 Aug 2008, our Solaris 10 system, which our admin tries to keep current regarding patches, does not have FEN. Since our application runs on Windows, Solaris and Linux, and since Windows and Linux both have this capability, it would be great if we could use a more or less constant design across all the platforms we support. FEN would make that possible. If you know anything about the status of FEN or when the Nevada build 72 is coming out, I hope you will respond to this post. Thanks for your help.
|
|
|
|
|
Posts:
5,504
From:
US
Registered:
3/9/05
|
|
|
|
Re: File events notification mechansim
Posted:
Aug 22, 2008 8:13 AM
in response to: gail
|
|
Gail Nagle wrote:
> The file event discussed here in May 2006 is close to what I am looking for. Its only shortcoming is that a separate event is needed for each directory being tracked. A more useful interface would allow events to be received for all create/delete activity in a directory tree, starting from a given node. In his blog entry dated 27 Aug 2007, Prakash noted that the functionality (FEN) would be available in the Nevada build 72. As of 20 Aug 2008, our Solaris 10 system, which our admin tries to keep current regarding patches, does not have FEN. Since our application runs on Windows, Solaris and Linux, and since Windows and Linux both have this capability, it would be great if we could use a more or less constant design across all the platforms we support. FEN would make that possible. If you know anything about the status of FEN or when the Nevada build 72 is coming out, I hope you will respond to this post. Thanks for your help.
Nevada is the release of Solaris after Solaris 10, currently
under development, but available in pre-release builds as
Solaris Express and OpenSolaris releases. (So your question
is sort of like saying "This feature was added in Vista, but
I don't see it in XP after applying all the XP hotfixes.")
You can download OpenSolaris & Solaris Express releases from:
http://www.opensolaris.org/os/downloads/
--
-Alan Coopersmith- alan dot coopersmith at sun dot com
Sun Microsystems, Inc. - X Window System Engineering
_______________________________________________
perf-discuss mailing list
perf-discuss at opensolaris dot org
|
|
|
|
|
Posts:
1
From:
Registered:
9/11/08
|
|
|
|
Re: File events notification mechansim
Posted:
Sep 11, 2008 7:36 AM
in response to: alanc
To: Communities » performance » discuss
|
|
Hi Alan,
So will the nevada release be an independent release of solaris after solaris 10 or will the feature of FEN be provided as some patch for solaris 10 as well.
Basically can we use this feature by being on solaris 10 itself or do we have to upgrade to a next version??
Currently for the FEN we need a thread polling the port by port_get to know the events.Will there be a support for signal based events so that threads do not have to keep looping for port_get and get the event notification which can later be used to do a port_get just once.
Also, besides FEN ,are any of the event completion framework features available on solaris 10 like " polling file descriptors", "user events" etc.?
Many thanks in advance!!!
|
|
|
|
|
