[CALUG] lug Digest, Vol 15, Issue 16
Tom Berlett
dragineez at yahoo.com
Wed Oct 25 11:43:46 CDT 2006
"I'm working on an application that has a really small footprint and can't
afford to run another database intance or provide any more table space in
any of the already-existing ones. I'm looking for something that will allow
me to implement a "/tmp"-like storage facility for miscellaneous data, but
in a single file."
Isn't that what XML is for? Plain text storage, small footprint, fast selects, inserts, deletes, and it's platform independent.
+-+-+-+-+-+-+-+-+
Motorcycle Roadracing - as a matter of fact it IS rocket science!
----- Original Message ----
From: "lug-request at calug.com" <lug-request at calug.com>
To: lug at calug.com
Sent: Wednesday, October 25, 2006 12:00:16 PM
Subject: lug Digest, Vol 15, Issue 16
Send lug mailing list submissions to
lug at calug.com
To subscribe or unsubscribe via the World Wide Web, visit
http://calug.com/mailman/listinfo/lug
or, via email, send a message with subject or body 'help' to
lug-request at calug.com
You can reach the person managing the list at
lug-owner at calug.com
When replying, please edit your Subject line so it is more specific
than "Re: Contents of lug digest..."
Today's Topics:
1. Looking for a tool (Jason C. Miller)
2. Re: Looking for a tool (Jason Dixon)
3. Re: Looking for a tool (Alan)
4. Re: Looking for a tool (Eric K. Dickinson)
5. Re: Looking for a tool (william kelly)
6. Re: Looking for a tool (Josiah Ritchie)
7. Re: Looking for a tool (Jim Sansing)
----------------------------------------------------------------------
Message: 1
Date: Wed, 25 Oct 2006 08:18:18 -0400
From: "Jason C. Miller" <jason.c.miller at gmail.com>
Subject: [CALUG] Looking for a tool
To: lug at calug.com
Message-ID:
<3803e6780610250518p446c8f1cm76aa3ecf450e9d7d at mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
I'm looking for a UNIX tool to perform a specific set of functionality but I
dont know if there's anything out there that would do the trick.
I need a very small and light-weight tool that will allow me to me to
"insert" any number of data into a plain file and provide an interface for
performing that "insert" and also "deletes" and "selects" (using database
terminology).
Confused yet? :)
I'm working on an application that has a really small footprint and can't
afford to run another database intance or provide any more table space in
any of the already-existing ones. I'm looking for something that will allow
me to implement a "/tmp"-like storage facility for miscellaneous data, but
in a single file.
-------------------------------------------------------------------------------
EXAMPLE: (using "STORE" as the storage filename)
Say I'm running a session-enabled website and need such a tool for the
task I'm describing above. Here's the kind of thing it would do...
$> sometool insert STORE "session id" "103847294837"
$> sometool insert STORE "number_users" "4"
$> sometool insert STORE "username" "foo"
$> sometool insert STORE "privs" "read/write"
$> sometool insert STORE "admin" "no"
$> SESSION=` sometool select STORE "session_id" `
$> echo ${SESSION}
103847294837
$> USER=` sometool select STORE "username" `
$> echo ${USER}
foo
-------------------------------------------------------------------------------
I need something that's just that simple. Just FYI...the session-enabled
website was stricly an example. It is NOT what I'm working on. It just
provided with a valid use-case for the tool. The solution I'm looking for
can't be OS-dependent (so Linux or Solaris filesystem tricks in leu of a
tool are out the window).
Any ideas? :)
-jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://calug.com/pipermail/lug/attachments/20061025/5836528f/attachment.html
------------------------------
Message: 2
Date: Wed, 25 Oct 2006 08:30:20 -0400
From: Jason Dixon <jason at dixongroup.net>
Subject: Re: [CALUG] Looking for a tool
To: Jason C.Miller <jason.c.miller at gmail.com>
Cc: lug at calug.com
Message-ID: <28D62028-AD97-4455-A044-AD5E6820D1CB at dixongroup.net>
Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed
On Oct 25, 2006, at 8:18 AM, Jason C. Miller wrote:
> I'm looking for a UNIX tool to perform a specific set of
> functionality but I
> dont know if there's anything out there that would do the trick.
I've been called a UNIX tool.
> I need a very small and light-weight tool that will allow me to me to
> "insert" any number of data into a plain file and provide an
> interface for
> performing that "insert" and also "deletes" and "selects" (using
> database
> terminology).
>
> Confused yet? :)
>
> I'm working on an application that has a really small footprint and
> can't
> afford to run another database intance or provide any more table
> space in
> any of the already-existing ones. I'm looking for something that
> will allow
> me to implement a "/tmp"-like storage facility for miscellaneous
> data, but
> in a single file.
>
> ----------------------------------------------------------------------
> ---------
> EXAMPLE: (using "STORE" as the storage filename)
> Say I'm running a session-enabled website and need such a tool
> for the
> task I'm describing above. Here's the kind of thing it would do...
>
> $> sometool insert STORE "session id" "103847294837"
> $> sometool insert STORE "number_users" "4"
> $> sometool insert STORE "username" "foo"
> $> sometool insert STORE "privs" "read/write"
> $> sometool insert STORE "admin" "no"
>
> $> SESSION=` sometool select STORE "session_id" `
> $> echo ${SESSION}
> 103847294837
> $> USER=` sometool select STORE "username" `
> $> echo ${USER}
> foo
> ----------------------------------------------------------------------
> ---------
>
> I need something that's just that simple. Just FYI...the session-
> enabled
> website was stricly an example. It is NOT what I'm working on. It
> just
> provided with a valid use-case for the tool. The solution I'm
> looking for
> can't be OS-dependent (so Linux or Solaris filesystem tricks in leu
> of a
> tool are out the window).
>
> Any ideas? :)
Nothing existing that I can think of. Could be done in a few hours
with either Perl or { sh + expect }. I think it would make more
sense in Perl, given the regex you'd need for SELECT statements.
--
Jason Dixon
DixonGroup Consulting
http://www.dixongroup.net
------------------------------
Message: 3
Date: Wed, 25 Oct 2006 08:39:48 -0400
From: "Alan" <5q2mxt102 at sneakemail.com>
Subject: Re: [CALUG] Looking for a tool
To: lug at calug.com
Message-ID: <27180-23561 at sneakemail.com>
Content-Type: text/plain; charset="iso-8859-1"
I suggest sqlite. It won't be a plain text file, but it is very lightweight
and allows DB terminology.
http://www.sqlite.org/
Mozilla projects are moving towards using sqlite and I know that Songbird
uses it.
On 10/25/06, Jason Dixon jason-at-dixongroup.net |CALUG| <
...> wrote:
>
> On Oct 25, 2006, at 8:18 AM, Jason C. Miller wrote:
>
> > I'm looking for a UNIX tool to perform a specific set of
> > functionality but I
> > dont know if there's anything out there that would do the trick.
>
> I've been called a UNIX tool.
>
> > I need a very small and light-weight tool that will allow me to me to
> > "insert" any number of data into a plain file and provide an
> > interface for
> > performing that "insert" and also "deletes" and "selects" (using
> > database
> > terminology).
> >
> > Confused yet? :)
> >
> > I'm working on an application that has a really small footprint and
> > can't
> > afford to run another database intance or provide any more table
> > space in
> > any of the already-existing ones. I'm looking for something that
> > will allow
> > me to implement a "/tmp"-like storage facility for miscellaneous
> > data, but
> > in a single file.
> >
> > ----------------------------------------------------------------------
> > ---------
> > EXAMPLE: (using "STORE" as the storage filename)
> > Say I'm running a session-enabled website and need such a tool
> > for the
> > task I'm describing above. Here's the kind of thing it would do...
> >
> > $> sometool insert STORE "session id" "103847294837"
> > $> sometool insert STORE "number_users" "4"
> > $> sometool insert STORE "username" "foo"
> > $> sometool insert STORE "privs" "read/write"
> > $> sometool insert STORE "admin" "no"
> >
> > $> SESSION=` sometool select STORE "session_id" `
> > $> echo ${SESSION}
> > 103847294837
> > $> USER=` sometool select STORE "username" `
> > $> echo ${USER}
> > foo
> > ----------------------------------------------------------------------
> > ---------
> >
> > I need something that's just that simple. Just FYI...the session-
> > enabled
> > website was stricly an example. It is NOT what I'm working on. It
> > just
> > provided with a valid use-case for the tool. The solution I'm
> > looking for
> > can't be OS-dependent (so Linux or Solaris filesystem tricks in leu
> > of a
> > tool are out the window).
> >
> > Any ideas? :)
>
> Nothing existing that I can think of. Could be done in a few hours
> with either Perl or { sh + expect }. I think it would make more
> sense in Perl, given the regex you'd need for SELECT statements.
>
> --
> Jason Dixon
> DixonGroup Consulting
> http://www.dixongroup.net
>
>
>
> _______________________________________________
> Columbia, Maryland Linux User's Group (CALUG) mailing list
> CALUG Website: http://www.calug.com
> Email postings to: lug at calug.com
> Change your list subscription options:
> http://calug.com/mailman/listinfo/lug
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://calug.com/pipermail/lug/attachments/20061025/2eb05e88/attachment.html
------------------------------
Message: 4
Date: Wed, 25 Oct 2006 08:49:01 -0400
From: "Eric K. Dickinson" <Eric.Dickinson at nih.gov>
Subject: Re: [CALUG] Looking for a tool
To: "Jason C. Miller" <jason.c.miller at gmail.com>
Cc: lug at calug.com
Message-ID: <453F5D3D.9090002 at nih.gov>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
sed?
http://www.grymoire.com/Unix/Sed.html
presuming the "plain file" is ASCII.
eric
Jason C. Miller wrote:
> I'm looking for a UNIX tool to perform a specific set of functionality but I
> dont know if there's anything out there that would do the trick.
>
> I need a very small and light-weight tool that will allow me to me to
> "insert" any number of data into a plain file and provide an interface for
> performing that "insert" and also "deletes" and "selects" (using database
> terminology).
>
> Confused yet? :)
>
> I'm working on an application that has a really small footprint and can't
> afford to run another database intance or provide any more table space in
> any of the already-existing ones. I'm looking for something that will allow
> me to implement a "/tmp"-like storage facility for miscellaneous data, but
> in a single file.
>
> -------------------------------------------------------------------------------
> EXAMPLE: (using "STORE" as the storage filename)
> Say I'm running a session-enabled website and need such a tool for the
> task I'm describing above. Here's the kind of thing it would do...
>
> $> sometool insert STORE "session id" "103847294837"
> $> sometool insert STORE "number_users" "4"
> $> sometool insert STORE "username" "foo"
> $> sometool insert STORE "privs" "read/write"
> $> sometool insert STORE "admin" "no"
>
> $> SESSION=` sometool select STORE "session_id" `
> $> echo ${SESSION}
> 103847294837
> $> USER=` sometool select STORE "username" `
> $> echo ${USER}
> foo
> -------------------------------------------------------------------------------
>
> I need something that's just that simple. Just FYI...the session-enabled
> website was stricly an example. It is NOT what I'm working on. It just
> provided with a valid use-case for the tool. The solution I'm looking for
> can't be OS-dependent (so Linux or Solaris filesystem tricks in leu of a
> tool are out the window).
>
> Any ideas? :)
>
> -jason
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: http://calug.com/pipermail/lug/attachments/20061025/5836528f/attachment.html
> _______________________________________________
> Columbia, Maryland Linux User's Group (CALUG) mailing list
> CALUG Website: http://www.calug.com
> Email postings to: lug at calug.com
> Change your list subscription options: http://calug.com/mailman/listinfo/lug
>
------------------------------
Message: 5
Date: Wed, 25 Oct 2006 09:57:39 -0400
From: "william kelly" <william.kelly.jr at gmail.com>
Subject: Re: [CALUG] Looking for a tool
To: "Eric K. Dickinson" <Eric.Dickinson at nih.gov>
Cc: lug at calug.com
Message-ID:
<1ad34df30610250657k33726bb2wadc60a26816e7c44 at mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
what database are you using? it should facilitate a tmp storage such as
mysql. also what program/scripting language are you using? php,perl? most
likely you want sqlite however the files are not flat.
http://us3.php.net/sqlite but that shouldnt be an issue because you can
always run another select.
William Kelly Jr
443-414-4549
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://calug.com/pipermail/lug/attachments/20061025/c0822af2/attachment.html
------------------------------
Message: 6
Date: Wed, 25 Oct 2006 10:31:12 -0400
From: "Josiah Ritchie" <josiah.ritchie at gmail.com>
Subject: Re: [CALUG] Looking for a tool
To: "Jason C. Miller" <jason.c.miller at gmail.com>
Cc: lug at calug.com
Message-ID:
<c5e95c230610250731i79417be6if9a5893e94537fbd at mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
On 10/25/06, Jason C. Miller <jason.c.miller at gmail.com> wrote:
> I'm looking for a UNIX tool to perform a specific set of functionality but I
> dont know if there's anything out there that would do the trick.
>
> I need a very small and light-weight tool that will allow me to me to
> "insert" any number of data into a plain file and provide an interface for
> performing that "insert" and also "deletes" and "selects" (using database
> terminology).
>
> Confused yet? :)
>
> I'm working on an application that has a really small footprint and can't
> afford to run another database intance or provide any more table space in
> any of the already-existing ones. I'm looking for something that will allow
> me to implement a "/tmp"-like storage facility for miscellaneous data, but
> in a single file.
>
> -------------------------------------------------------------------------------
> EXAMPLE: (using "STORE" as the storage filename)
> Say I'm running a session-enabled website and need such a tool for the
> task I'm describing above. Here's the kind of thing it would do...
>
> $> sometool insert STORE "session id" "103847294837"
> $> sometool insert STORE "number_users" "4"
> $> sometool insert STORE "username" "foo"
> $> sometool insert STORE "privs" "read/write"
> $> sometool insert STORE "admin" "no"
>
> $> SESSION=` sometool select STORE "session_id" `
> $> echo ${SESSION}
> 103847294837
> $> USER=` sometool select STORE "username" `
> $> echo ${USER}
> foo
> -------------------------------------------------------------------------------
>
> I need something that's just that simple. Just FYI...the session-enabled
> website was stricly an example. It is NOT what I'm working on. It just
> provided with a valid use-case for the tool. The solution I'm looking for
> can't be OS-dependent (so Linux or Solaris filesystem tricks in leu of a
> tool are out the window).
>
> Any ideas? :)
I know it doesn't quite fit the SQL syntax, but maybe rrdtool is worth
exploring? In truth I don't know much about it, but sounds like the
closest thing I can think of to what you're doing.
JSR/
--
Our Mission
Technology and Hospitality for God's Workmen
http://missions.ritchietribe.net
------------------------------
Message: 7
Date: Wed, 25 Oct 2006 12:04:01 -0400
From: Jim Sansing <jjsansing at comcast.net>
Subject: Re: [CALUG] Looking for a tool
To: lug at calug.com
Message-ID: <453F8AF1.3030208 at comcast.net>
Content-Type: text/plain; charset=ISO-8859-1
What you are describing is the VSAM file system on the IBM
mainframe MVS operating system. It has a 'count/key/data'
structure that allows you to find specific variable length records
quickly. I did a little searching and couldn't find any library
that would support that for you on a Linux/UNIX platform.
However, it shouldn't be too hard to implement. The first field
of each record is the size, the second is the key, and the third is
the value. To search the file, you either do a list search--you
don't have to read every byte since you know the size--or you
maintain an index in memory (now you're into sorting for which
I recommend a Red Black tree, see:
http://web.mit.edu/~emin/www/source_code/index.html).
To do deletes, the brute force method would be to define a
'deleted' key and periodically defragment it by overlaying
deleted records. If you know the max size of your records,
mmap should do this efficiently with 2 fixed size buffers.
If you think this will require more resources than you have,
any tool will be using at least the same without your knowing
about it. At least the roll-your-own method gives you control
so you can eliminate unnecessary pieces (otherwise known as
cutting corners ;~).
The only other option I know of for Linux/UNIX is Berkeley DB
which uses flat files. Subversion uses it, so you could check out
their code as an example.
Later . . . Jim
Jason C. Miller wrote:
>I'm looking for a UNIX tool to perform a specific set of functionality but I
>dont know if there's anything out there that would do the trick.
>
>I need a very small and light-weight tool that will allow me to me to
>"insert" any number of data into a plain file and provide an interface for
>performing that "insert" and also "deletes" and "selects" (using database
>terminology).
>
>Confused yet? :)
>
>I'm working on an application that has a really small footprint and can't
>afford to run another database intance or provide any more table space in
>any of the already-existing ones. I'm looking for something that will allow
>me to implement a "/tmp"-like storage facility for miscellaneous data, but
>in a single file.
>
>-------------------------------------------------------------------------------
>EXAMPLE: (using "STORE" as the storage filename)
> Say I'm running a session-enabled website and need such a tool for the
>task I'm describing above. Here's the kind of thing it would do...
>
>$> sometool insert STORE "session id" "103847294837"
>$> sometool insert STORE "number_users" "4"
>$> sometool insert STORE "username" "foo"
>$> sometool insert STORE "privs" "read/write"
>$> sometool insert STORE "admin" "no"
>
>$> SESSION=` sometool select STORE "session_id" `
>$> echo ${SESSION}
>103847294837
>$> USER=` sometool select STORE "username" `
>$> echo ${USER}
>foo
>-------------------------------------------------------------------------------
>
>I need something that's just that simple. Just FYI...the session-enabled
>website was stricly an example. It is NOT what I'm working on. It just
>provided with a valid use-case for the tool. The solution I'm looking for
>can't be OS-dependent (so Linux or Solaris filesystem tricks in leu of a
>tool are out the window).
>
>Any ideas? :)
>
> -jason
>-------------- next part --------------
>An HTML attachment was scrubbed...
>URL: http://calug.com/pipermail/lug/attachments/20061025/5836528f/attachment.html
>_______________________________________________
>Columbia, Maryland Linux User's Group (CALUG) mailing list
>CALUG Website: http://www.calug.com
>Email postings to: lug at calug.com
>Change your list subscription options: http://calug.com/mailman/listinfo/lug
>
>
------------------------------
_______________________________________________
lug mailing list
lug at calug.com
http://calug.com/mailman/listinfo/lug
End of lug Digest, Vol 15, Issue 16
***********************************
More information about the lug
mailing list