|
|
flag day: extended FILE standard I/O
Date: Thu, 20 Apr 2006 08:51:58 -0700 (PDT)
From: Craig Mohrman <Craig dot Mohrman at eng dot sun dot com>
To: on-all at sun dot com, onnv-gate at onnv dot eng dot sun dot com
Subject: flag day: extended FILE standard I/O
--Drey_of_Squirrels_426_000
Content-Type: TEXT/plain; charset=us-ascii
Content-MD5: WwPZO8YbtQo273Lh39M8AA==
This message will be repeated to all software engineering.
Greetings,
If you are not a Solaris software engineer then you can skip this message.
We have finally fixed:
1085341 32-bit stdio routines should support file descriptors >255
We have just putback this change into Solaris Nevada build 39.
We have both runtime solutions that will be back-ported to Solaris 9 and 10
as well as programming solutions to fix code. This fix is only relevant
in the 32-bit world. The 64-bit world has never had this issue.
The main reason for this message is to alert you that your code will
not compile if you are directly referencing "the field formerly known
as _file" of the FILE structure when you:
#include <stdio.h>
If your code is simply reading the value of _file, you should use
fileno(3C), since the value found in "the field formerly known as _file"
might no longer contain the FILE's file descriptor.
int fd = fileno(fileptr);
You should no longer assign a new value to _file.
To use the extended FILE feature you must first raise the file
descriptor limit, either from the shell or programatically, since
the default limit for file descriptors remains at 256.
If you want to programmatically use extended FILE facility, you need only add
"F" to fopen(3C) as the last mode character.
FILE *ptr = fopen("filename", "rF");
^
Please read the attached fopen(3C) man page for restrictions.
You must not return this extended FILE handle to higher level code
that might not know how to handle it. Optimally, you would compile all
source code to reveal any improper access to FILE structure members.
If you work on library code or plugable objects that open files using
standard I/O, such code might currently fail when file descriptors under 256
have been exhausted. This code should now be fixed by appending an "F"
to the fopen() flags, except where the FILE *s are returned to the caller.
A shadow standard I/O implementation exists in libnsl to deal with
this problem (__nsl_*). No new code should call those functions;
they will be removed in a follow-up putback.
The "F" must not be used if the FILE *fptr is to be returned to a caller.
The calling application might not understand how to handle it.
Instead you might consider using enable_extended_FILE_stdio(3C)
at a higher level in the code.
For binary relief please read extendedFILE.5.
It is necessary to use only one of the methods described above.
I've attached drafts of all the relevant man pages so you can start
learning now.
I've also attached a couple of coding examples.
Please pass this message to any field engineers or other parties
that might be interested.
Thanks for reading and enjoy your extended FILE's.
craig dot mohrman at sun dot com
casper dot dik at sun dot com
don dot cragun at sun dot com
--Drey_of_Squirrels_426_000
Content-Type: TEXT/plain; name="enable_extended_FILE_stdio.text"; charset=us-ascii; x-unix-mode=0644
Content-Description: enable_extended_FILE_stdio.text
Content-MD5: JBQWwx1gEli7yr/UBcpupA==
Standard C Library Functions enable_extended_FILE_stdio(3C)
NAME
enable_extended_FILE_stdio - enable extended FILE facility
within standard I/O
SYNOPSIS
#include <stdio.h>
#include <stdio_ext.h>
#include <signal.h>
int enable_extended_FILE_stdio(int low_fd, int
signal_action);
DESCRIPTION
The enable_extended_FILE_stdio() function enables the use of
the extended FILE facility (see NOTES) and determines which,
if any, signal will be sent when an application uses FILE-
>_file inappropriately.
The low_fd argument specifies the lowest file descriptor in
the range 3 through 255 that the application wants to be
selected as the unallocatable file descriptor. File
descriptors 0, 1, and 2 cannot be used because they are
reserved for use as the default file descriptors underlying
the stdin, stdout, and stderr standard I/O streams. The
low_fd argument can also be set to -1 to request that
enable_extended_FILE_stdio() select a "reasonable" unallo-
catable file descriptor. In this case,
enable_extended_FILE_stdio() will first attempt to reserve a
relatively large file descriptor, but will keep trying to
find an unallocatable file descriptor until it is known that
no file descriptor can be reserved.
The signal_action argument specifies the signal that will be
sent to the process when the unallocatable file descriptor
is used as a file descriptor argument to any system call
except close(2). If signal_action is -1, the default signal
(SIGABRT) will be sent. If signal_action is 0, no signal
will be sent. Otherwise, the signal specified by
signal_action will be sent.
The enable_extended_FILE_stdio() function calls
unallocatablefd = fcntl(low_fd, F_BADFD, action);
to reserve the unallocatable file descriptor and set the
signal to be sent if the unallocatable file descriptor is
used in a system call. If the fcntl(2) call succeeds, the
extended FILE facility is enabled and the unallocatable file
descriptor is saved for later use by the standard I/O func-
tions. When an attempt is made to open a standard I/O
stream (see fdopen(3C), fopen(3C), and popen(3C)) with an
underlying file descriptor greater than 255, the file
descriptor is stored in an auxiliary location and the field
formerly known as FILE->_file is set to the unallocatable
file descriptor.
If the file descriptor limit for the process is less than or
equal to 256 (the system default), the application needs to
raise the limit (see getrlimit(2)) for the extended FILE
facility to be useful. The enable_extended_FILE_stdio()
function does not attempt to change the file descriptor
limit.
This function is used by the extendedFILE(5) preloadable
library to enable the extended FILE facility.
RETURN VALUES
Upon successful completion, enable_extended_FILE_stdio()
returns 0. Otherwise, -1 is returned and errno is set to
indicate the error.
ERRORS
The enable_extended_FILE_stdio() function will fail if:
EAGAIN All file descriptors in the inclusive range
3 through 255 refer to files that are
currently open in the process.
EBADF The low_fd argument is greater than 255, or
is less than 3 and not equal to -1.
EEXIST A file descriptor has already been marked by
an earlier call to fcntl().
EINVAL The signal_action argument is not -1, is not
0, and is not a valid signal number.
USAGE
The enable_extended_FILE_stdio() function is available only
in the 32-bit compilation environment.
The fdopen(3C), fopen(3C), and popen(3C) functions all
enable the use of the extended FILE facility. For source
changes, a trailing F character in the mode argument can be
used with any of these functions if the FILE *fptr is used
only within the context of a single function or group of
functions and not meant to be returned to a caller. All of
the source code to the application must then be recompiled,
thereby exposing any improper usage of the FILE structure
fields.
The F character must not be used if the FILE *fptr is to be
returned to a caller. The calling application might not
understand how to process it. Alternatively, the
enable_extended_FILE_stdio() function can be used at a
higher level in the code.
Use extendedFILE(5) for binary relief.
EXAMPLES
Example 1: Increase the file limit and enable extended FILE
facility.
The following example demonstrates how to programmatically
increase the file limit and enable extended FILE facility.
(void) getrlimit(RLIMIT_NOFILE, &rlp);
rlp.rlim_cur = 1000; /* set the desired number of file descriptors */
retval = setrlimit(RLIMIT_NOFILE, &lrp);
if (retval == -1) {
/* error */
}
/* enable extended FILE facility */
retval = enable_extended_FILE_stdio(-1, SIGABRT);
if (retval == -1) {
/* error */
}
ATTRIBUTES
See attributes(5) for descriptions of the following attri-
butes:
____________________________________________________________
| ATTRIBUTE TYPE | ATTRIBUTE VALUE |
|_____________________________|_____________________________|
| Availability | SUNWcsl (32-bit) |
|_____________________________|_____________________________|
| Interface Stability | Evolving |
|_____________________________|_____________________________|
| MT-Level | Safe |
|_____________________________|_____________________________|
SEE ALSO
close(2), fcntl(2), getrlimit(2), extendedFILE(3LIB),
fdopen(3C), fopen(3C), popen(3C), sihnal.h(3HEAD),
stdio(3C), attributes(5), extendedFILE(5)
NOTES
Historically, 32-bit Solaris applications have been limited
to using only the file descriptors 0 through 255 with the
standard I/O functions (see stdio(3C)) in the C library. The
extended FILE facility allows well-behaved 32-bit applica-
tions to use any valid file descriptor with the standard I/O
functions.
For the purposes of the extended FILE facility, a well-
behaved application is one that:
o does not directly access any fields in the FILE struc-
ture pointed to by the FILE pointer associated with any
standard I/O stream,
o checks all return values from standard I/O functions
for error conditions, and
o behaves appropriately when an error condition is
reported.
The extended FILE facility generates EBADF error returns and
optionally delivers a signal to the calling process on most
attempts to use the file descriptor formerly stored in
FILE->_file as an argument to a system call when a file
descriptor value greater than 255 is being used to access
the file underlying the corresponding FILE pointer. The
only exception is that calls to the close() system call will
return an EBADF error in this case, but will not deliver the
signal. The FILE->_file has been renamed to help applica-
tions quickly detect code that needs to be updated.
The extended FILE facility should only be used by well-
behaved applications. Although the extended FILE facility
reports errors, applications that directly reference FILE-
>_file should be updated to use public interfaces rather
than rely on implementation details that no longer work as
the application expects (see __fbufsize(3C) and fileno(3C).
This facility takes great care to avoid problems in well-
behaved applications while maintaining maximum compatibil-
ity. It also attempts to catch dangerous behavior in appli-
cations that are not well-behaved as soon as possible and to
notify those applications as soon as bad behavior is
detected.
There are, however, limitations. For example, if an appli-
cation enables this facility and is linked with an object
file that had a standard I/O stream using an extended FILE
pointer, and then used the sequence
(void) close(FILE->_file);
FILE->_file = myfd;
to attempt to change the file descriptor associated with the
stream, undesired results can occur. The close() function
will fail, but since this usage ignores the return status,
the application proceeds to perform low level I/O on FILE-
>_file while calls to standard I/O functions would continue
to use the original, extended FILE pointer. If the
application continues using standard I/O functions after
changing FILE->_file, silent data corruption could occur
because the application thinks it has changed file descrip-
tors with the above assignment but the actual standard I/O
file descriptor is stored in the auxiliary location. The
chances for corruption are even higher if myfd has a value
greater than 255 and is truncated by the assignment to the
8-bit _file field.
Since the _file field has been renamed, attempts to recom-
pile this code will fail. The application should be changed
not to use this field in the FILE structure.
The application should not use this facility if it uses
_file directly, including using the fileno() macro that was
provided in stdio.h(3HEAD) in Solaris 2.0 through 2.7.
--Drey_of_Squirrels_426_000
Content-Type: TEXT/plain; name="extendedFILE.text"; charset=us-ascii; x-unix-mode=0644
Content-Description: extendedFILE.text
Content-MD5: 4jKxsX/XC40S/yIxKh91kA==
Interface Libraries extendedFILE(5)
NAME
extendedFILE - enable extended FILE facility usage
SYNOPSIS
$ ulimit -n N_file_descriptors
$ LD_PRELOAD_32=/usr/lib/extendedFILE.so.1 application [arg...]
DESCRIPTION
The extendedFILE.so.1 is not a library but an enabler of the
extended FILE facility.
The extended FILE facility allows 32-bit processes to use
any valid file descriptor with the standard I/O (see
stdio(3C)) C library functions. Historically, 32-bit appli-
cations have been limited to using the first 256 numerical
file descriptors for use with standard I/O streams. By
using the extended FILE facility this limitation is lifted.
Any valid file descriptor can be used with standard I/O. See
the NOTES section of enable_extended_FILE_stdio(3C).
The extended FILE facility is enabled from the shell level
before an application is launched. The file descriptor limit
must also be raised. The syntax for raising the file
descriptor limit is
$ ulimit -n max_file_descriptors
$ LD_PRELOAD_32=/usr/lib/extendedFILE.so.1 application [arg...]
where max_file_descriptors is the maximum number of file
descriptors desired. See limit(1). The maximum value is the
same as the maximum value for open(2).
ENVIRONMENT VARIABLES
The following environment variables control the behavior of
the extended FILE facility.
_STDIO_BADFD This variable takes an integer
representing the lowest file
descriptor, which will be made unal-
locatable. This action provides a
protection mechanism so that appli-
cations that abuse interfaces do not
experience silent data corruption.
The value must be between 3 and 255
inclusive.
_STDIO_BADFD_SIGNAL This variable takes an integer or
string representing any valid sig-
nal. See signal.h(3HEAD) for valid
values or strings. This environment
variable causes the specified signal
to be sent to the application if
certain exceptional cases are
detected during the use of this
facility. The default signal is
SIGABRT.
EXAMPLES
Example 1: Limit the number of file descriptors and FILE
standard I/O structures.
The following example limits the number of file descriptors
and FILE standard I/O structures to 1000.
$ ulimit -n 1000
$ LD_PRELOAD_32=/usr/lib/extendedFILE.so.1 application [arg...]
Example 2: Enable the extended FILE facility.
The following example enables the extended FILE facility.
See enable_extended_FILE_stdio(3C) for more examples.
$ ulimit -n 1000
$ _STDIO_BADFD=100 _STDIO_BADFD_SIGNAL=SIGABRT \
LD_PRELOAD_32=/usr/lib/extendedFILE.so.1 application [arg ...]
Example 3: Set up the extended FILE environment and start
the application.
The following shell script first sets up the proper extended
FILE environment and then starts the application:
#!/bin/sh
if [ $# = 0 ]; then
echo "usage: $0 application [arguments...]"
exit 1
fi
ulimit -n 1000
# _STDIO_BADFD=196; export _STDIO_BADFD
# _STDIO_BADFD_SIGNAL=SIGABRT; export _STDIO_BADFD_SIGNAL
LD_PRELOAD_32=/usr/lib/extendedFILE.so.1; export LD_PRELOAD_32
"$@"
ATTRIBUTES
See attributes(5) for descriptions of the following attri-
butes:
____________________________________________________________
| ATTRIBUTE TYPE | ATTRIBUTE VALUE |
|_____________________________|_____________________________|
| Availability | SUNWcsl (32-bit) |
|_____________________________|_____________________________|
| Interface Stability | Stable |
|_____________________________|_____________________________|
| MT-Level | Safe |
|_____________________________|_____________________________|
SEE ALSO
limit(1), open(2), enable_extended_FILE_stdio(3C),
fdopen(3C), fopen(3C), popen(3C), signal.h(3HEAD),
stdio(3C), attributes(5)
WARNINGS
The following displayed message
Application violated extended FILE safety mechanism.
Please read the man page for extendedFILE.
Aborting
is an indication that your application is modifying the
internal file descriptor field of the FILE structure from
standard I/O. Continued use of this extended FILE facility
could harm your data. Do not use the extended FILE facility
with your application.
--Drey_of_Squirrels_426_000
Content-Type: TEXT/plain; name="fopen.text"; charset=us-ascii; x-unix-mode=0644
Content-Description: fopen.text
Content-MD5: vvWWdnNkq0eWlsuWudaAuw==
Standard C Library Functions fopen(3C)
NAME
fopen - open a stream
SYNOPSIS
#include <stdio.h>
FILE *fopen(const char *filename, const char *mode);
DESCRIPTION
The fopen() function opens the file whose pathname is the
string pointed to by filename, and associates a stream with
it.
The argument mode points to a string beginning with one of
the following sequences:
r or rb Open file for reading.
w or wb Truncate to zero length or create
file for writing.
a or ab Append; open or create file for
writing at end-of-file.
r+ or rb+ or r+b Open file for update (reading and
writing).
w+ or wb+ or w+b Truncate to zero length or create
file for update.
a+ or ab+ or a+b Append; open or create file for
update, writing at end-of-file.
The character b has no effect, but is allowed for ISO C
standard conformance (see standards(5)). Opening a file with
read mode (r as the first character in the mode argument)
fails if the file does not exist or cannot be read.
Opening a file with append mode (a as the first character in
the mode argument) causes all subsequent writes to the file
to be forced to the then current end-of-file, regardless of
intervening calls to fseek(3C). If two separate processes
open the same file for append, each process may write freely
to the file without fear of destroying output being written
by the other. The output from the two processes will be
intermixed in the file in the order in which it is written.
When a file is opened with update mode (+ as the second or
third character in the mode argument), both input and output
may be performed on the associated stream. However, output
must not be directly followed by input without an interven-
ing call to fflush(3C) or to a file positioning function (
fseek(3C), fsetpos(3C) or rewind(3C)), and input must not be
directly followed by output without an intervening call to a
file positioning function, unless the input operation
encounters end-of-file.
When opened, a stream is fully buffered if and only if it
can be determined not to refer to an interactive device. The
error and end-of-file indicators for the stream are cleared.
If mode begins with w or a and the file did not previously
exist, upon successful completion, fopen() function will
mark for update the st_atime, st_ctime and st_mtime fields
of the file and the st_ctime and st_mtime fields of the
parent directory.
If mode begins with w and the file did previously exist,
upon successful completion, fopen() will mark for update the
st_ctime and st_mtime fields of the file. The fopen() func-
tion will allocate a file descriptor as open(2) does.
Normally, 32-bit applications return an EMFILE error when
attempting to associate a stream with a file accessed by a
file descriptor with a value greater than 255. If the last
character of mode is F, 32-bit applications will be allowed
to associate a stream with a file accessed by a file
descriptor with a value greater than 255. A FILE pointer
obtained in this way must never be used by any code that
might directly access fields in the FILE structure. If the
fields in the FILE structure are used directly by 32-bit
applications when the last character of mode is F, data
corruption could occur. See the USAGE section of this manual
page and the enable_extended_FILE_stdio(3C) manual page for
other options for enabling the extended FILE facility.
In 64-bit applications, the last character of mode is
silently ignored if it is F. 64-bit applications are always
allowed to associate a stream with a file accessed by a file
descriptor with any value.
The largest value that can be represented correctly in an
object of type off_t will be established as the offset max-
imum in the open file description.
RETURN VALUES
Upon successful completion, fopen() returns a pointer to the
object controlling the stream. Otherwise, a null pointer is
returned and errno is set to indicate the error.
The fopen() function may fail and not set errno if there are
no free stdio streams.
ERRORS
The fopen() function will fail if:
EACCES Search permission is denied on a
component of the path prefix, or the
file exists and the permissions
specified by mode are denied, or the
file does not exist and write per-
mission is denied for the parent
directory of the file to be created.
EINTR A signal was caught during the exe-
cution of fopen().
EISDIR The named file is a directory and
mode requires write access.
ELOOP Too many symbolic links were encoun-
tered in resolving path.
EMFILE There are {OPEN_MAX} file descrip-
tors currently open in the calling
process.
ENAMETOOLONG The length of the filename exceeds
PATH_MAX or a pathname component is
longer than NAME_MAX.
ENFILE The maximum allowable number of
files is currently open in the sys-
system.
ENOENT A component of filename does not
name an existing file or filename is
an empty string.
ENOSPC The directory or file system that
would contain the new file cannot be
expanded, the file does not exist,
and it was to be created.
ENOTDIR A component of the path prefix is
not a directory.
ENXIO The named file is a character spe-
cial or block special file, and the
device associated with this special
file does not exist.
EOVERFLOW The current value of the file posi-
tion cannot be represented correctly
in an object of type fpos_t.
EROFS The named file resides on a read-
only file system and mode requires
write access.
The fopen() function may fail if:
EINVAL The value of the mode argument is
not valid.
EMFILE {FOPEN_MAX} streams are currently
open in the calling process.
{STREAM_MAX} streams are currently
open in the calling process.
ENAMETOOLONG Pathname resolution of a symbolic
link produced an intermediate result
whose length exceeds {PATH_MAX}.
ENOMEM Insufficient storage space is avail-
able.
ETXTBSY The file is a pure procedure (shared
text) file that is being executed
and mode requires write access.
USAGE
A process is allowed to have at least {FOPEN_MAX} stdio
streams open at a time. For 32-bit applications, however,
the underlying ABIs formerly required that no file descrip-
tor used to access the file underlying a stdio stream have a
value greater than 255. To maintain binary compatibility
with earlier Solaris releases, this limit still constrains
32-bit applications. However, when a 32-bit application is
aware that no code that has access to the FILE pointer
returned by fopen() will use the FILE pointer to directly
access any fields in the FILE structure, the F character can
be used as the last character in the mode argument to cir-
cumvent this limit. Because it could lead to data corrup-
tion, the F character in mode must never be used when the
FILE pointer might later be used by binary code unknown to
the user. The F character in mode is intended to be used by
library functions that need a FILE pointer to access data to
process a user request, but do not need to pass the FILE
pointer back to the user. 32-bit applications that have been
inspected can use the extended FILE facility to circumvent
this limit if the inspection shows that no FILE pointers
will be used to directly access FILE structure contents.
The fopen() function has a transitional interface for 64-bit
file offsets. See lf64(5).
ATTRIBUTES
See attributes(5) for descriptions of the following attri-
butes:
____________________________________________________________
| ATTRIBUTE TYPE | ATTRIBUTE VALUE |
|_____________________________|_____________________________|
| Interface Stability | See below. |
|_____________________________|_____________________________|
| MT-Level | MT-Safe |
|_____________________________|_____________________________|
The F character in the mode argument is Evolving. In all
other respects this function is Standard.
SEE ALSO
enable_extended_FILE_stdio(3C), fclose(3C), fdopen(3C),
fflush(3C), freopen(3C), fsetpos(3C), rewind(3C), attri-
butes(5), lf64(5), standards(5)
--Drey_of_Squirrels_426_000
Content-Type: TEXT/x-sun-c-file; name="fopenF.c"; charset=us-ascii; x-unix-mode=0644
Content-Description: fopenF.c
Content-MD5: QwkS8tw81eOT/EwgW098+A==
/*
cc -v -g -o fopenF fopenF.c
Raise the file descriptor limit:
csh% limit descriptors 600
ksh$ ulimit -n 600
To see binary relief:
% env LD_PRELOAD_32=/usr/lib/extendedFILE.so.1 fopenF 400
To see programmatic relief:
% fopenF 500 F
*/
#include <stdio.h>
#include <stdlib.h>
int
main(int argc, char *argv[])
{
FILE *fptr;
int n_fopens;
int i;
int Fflag = 0;
if (argc == 1)
{
fprintf(stderr, "usage: %s n_fopens [\"F\" for flag]\n",
argv[0]);
exit(1);
} else
n_fopens = atoi(argv[1]);
if (argc == 3)
Fflag = 1;
for (i = 0; i < n_fopens; i++)
{
fptr = fopen("/tmp/fopenF_test", Fflag ? "wF" : "w");
if (fptr == NULL)
{
fprintf(stderr, "fopen failed\n");
exit(1);
}
printf("fd = %d\n", fileno(fptr));
}
return(0);
}
--Drey_of_Squirrels_426_000
Content-Type: TEXT/x-sun-c-file; name="enable.c"; charset=us-ascii; x-unix-mode=0644
Content-Description: enable.c
Content-MD5: 10RHaOlBcDCCKKfsWVweWQ==
/*
cc -v -g -o enable enable.c
Raise the file descriptor limit:
csh% limit descriptors 600
ksh$ ulimit -n 600
To programmatically do what /usr/lib/extendedFILE.so.1 does:
% enable 500
*/
#include <stdio.h>
#include <stdio_ext.h>
#include <stdlib.h>
int
main(int argc, char *argv[])
{
FILE *fptr;
int n_fopens;
int i;
if (argc == 1)
{
fprintf(stderr, "usage: %s n_fopens\n",
argv[0]);
exit(1);
} else
n_fopens = atoi(argv[1]);
if (enable_extended_FILE_stdio(-1, -1) == -1)
{
fprintf(stderr, "enable_extended_FILE_stdio failed\n");
exit(1);
}
for (i = 0; i < n_fopens; i++)
{
fptr = fopen("/tmp/enable_test", "w");
if (fptr == NULL)
{
fprintf(stderr, "fopen failed\n");
exit(1);
}
printf("fd = %d\n", fileno(fptr));
}
return(0);
}
--Drey_of_Squirrels_426_000--
|