using Line Input

2 Mar 2007 4:58 PM

Mike

Hi,

I have been trying to use Line Input to read a file line by line

MS e.g
Dim TextLine
Open CommonDialog1.FileName For Input As #1   ' Open file.
Do While Not EOF(1)   ' Loop until end of file.
   Line Input #1, TextLine   ' Read line into variable.
   Debug.Print TextLine   ' Print to the Immediate window.
Loop
Close #1   ' Close file.
My problem is that although these files are mostly plain text - I just
usually use notepad to work with them they contain 'substitute' characters
(0x1A) and line input stops when it encounters one.How can I read this file
in VB6 line at a time? All I need is the plain text.I have triedDim fnum As
Long
Dim bytvar As Byte
Dim getValue As String
fnum = FreeFile()
Open CommonDialog1.FileName For Binary As #fnum
While Not EOF(fnum)
    Get #fnum, , bytvar
    Debug.Print Chr(bytvar)
Wend
Close #fnum

But it is slow and I would have to search for a crlf in order to know when I
have reached the end of the line wouldn't I?

thanks

Mike

2 Mar 2007 5:20 PM

Rick Rothstein (MVP - VB)

Show quote Hide quote

> I have been trying to use Line Input to read a file line by line
>
> MS e.g
> Dim TextLine
> Open CommonDialog1.FileName For Input As #1   ' Open file.
> Do While Not EOF(1)   ' Loop until end of file.
>   Line Input #1, TextLine   ' Read line into variable.
>   Debug.Print TextLine   ' Print to the Immediate window.
> Loop
> Close #1   ' Close file.
> My problem is that although these files are mostly plain text - I just
> usually use notepad to work with them they contain 'substitute' characters
> (0x1A) and line input stops when it encounters one.How can I read this
> file in VB6 line at a time? All I need is the plain text.I have triedDim
> fnum As Long
> Dim bytvar As Byte
> Dim getValue As String
> fnum = FreeFile()
> Open CommonDialog1.FileName For Binary As #fnum
> While Not EOF(fnum)
>    Get #fnum, , bytvar
>    Debug.Print Chr(bytvar)
> Wend
> Close #fnum
>
> But it is slow and I would have to search for a crlf in order to know when
> I have reached the end of the line wouldn't I?

Instead of trying to process the file piecemeal, I would just read the whole
file in, split it at the newline character sequences and then process the
individual lines from the array that was created (unless your file is
extremely large; say, over 30 or 40 Megs). For example....

fnum = FreeFile
Open CommonDialog1.FileName For Binary As fnum
    TotalFile = Space(LOF(fnum))
    Get fnum, , TotalFile
Close fnum
Records = Split(TotalFile, vbCrLf)
' Assuming you don't need the total file in memory
' once you have split out the individual lines
TotalFile = ""
For X = 0 To UBound(Records)
    Debug.Print Records(X)
Next
' When you are finally through with the array
Erase Records

The above assumes you have Dim'med the variables appropriately.

Rick

2 Mar 2007 5:40 PM

Mike

Hi,

The files can be large and the machine that the job will be run on is itself
ratther slow - that is why I didn't want to read the whole file and Split
it. Thanks for the suggestion though, perhaps this is the only way to do it
though?

Mike
Show quoteHide quote

> Instead of trying to process the file piecemeal, I would just read the
> whole file in, split it at the newline character sequences and then
> process the individual lines from the array that was created (unless your
> file is extremely large; say, over 30 or 40 Megs). For example....
>
> fnum = FreeFile
> Open CommonDialog1.FileName For Binary As fnum
>    TotalFile = Space(LOF(fnum))
>    Get fnum, , TotalFile
> Close fnum
> Records = Split(TotalFile, vbCrLf)
> ' Assuming you don't need the total file in memory
> ' once you have split out the individual lines
> TotalFile = ""
> For X = 0 To UBound(Records)
>    Debug.Print Records(X)
> Next
> ' When you are finally through with the array
> Erase Records
>
> The above assumes you have Dim'med the variables appropriately.
>
> Rick
>

2 Mar 2007 5:52 PM

Robert Morley

It gets a bit more tricky, but you can try reading the file in chunks and
splitting each of the chunks. The tricky part (okay, not THAT tricky) is in
making sure that you combine the last string in one chunk with the first
string in the next chunk. To my knowledge, Line Input works great on
"normal" text files, but if you want to step outside of that, you'll have
problems. Manual parsing is probably how you're going to have to do it,
unless maybe there's a third-party DLL that works better; I've never looked
at any, so I wouldn't know.

Rob

Show quoteHide quote

"Mike" <m***@n-o--s-p-a-m.luusac.co.uk> wrote in message
news:skZFh.38394$z54.12265@newsfe3-gui.ntli.net...
> Hi,
>
> The files can be large and the machine that the job will be run on is
> itself ratther slow - that is why I didn't want to read the whole file and
> Split it. Thanks for the suggestion though, perhaps this is the only way
> to do it though?
>
> Mike
>> Instead of trying to process the file piecemeal, I would just read the
>> whole file in, split it at the newline character sequences and then
>> process the individual lines from the array that was created (unless your
>> file is extremely large; say, over 30 or 40 Megs). For example....
>>
>> fnum = FreeFile
>> Open CommonDialog1.FileName For Binary As fnum
>>    TotalFile = Space(LOF(fnum))
>>    Get fnum, , TotalFile
>> Close fnum
>> Records = Split(TotalFile, vbCrLf)
>> ' Assuming you don't need the total file in memory
>> ' once you have split out the individual lines
>> TotalFile = ""
>> For X = 0 To UBound(Records)
>>    Debug.Print Records(X)
>> Next
>> ' When you are finally through with the array
>> Erase Records
>>
>> The above assumes you have Dim'med the variables appropriately.
>>
>> Rick
>>
>
>

3 Mar 2007 8:40 AM

J French

On Fri, 2 Mar 2007 12:52:25 -0500, "Robert Morley"
<rmor***@magma.ca.N0.Freak1n.sparn> wrote:

>It gets a bit more tricky, but you can try reading the file in chunks and
>splitting each of the chunks. The tricky part (okay, not THAT tricky) is in
>making sure that you combine the last string in one chunk with the first
>string in the next chunk. To my knowledge, Line Input works great on
>"normal" text files, but if you want to step outside of that, you'll have
>problems. Manual parsing is probably how you're going to have to do it,
>unless maybe there's a third-party DLL that works better; I've never looked
>at any, so I wouldn't know.

Here is a Class that encapsulates what Robert is suggesting.

Save it as cRFStrm.cls

Usage:
    Dim oRFS As cReadFileStream

    Set oRFS = New cReadFileStream
    If oRFS.Create( "c:\test.txt" ) = False Then
       ' bug out
    End If

    While oRFS.EofFlag = False
              L$ = oRFS.ReadDelineatedLine
              ....
    Wend
    oRFS.Free    (Optional)

VERSION 1.0 CLASS
BEGIN
MultiUse = -1 'True
END
Attribute VB_Name = "cReadFileStream"
Attribute VB_GlobalNameSpace = False
Attribute VB_Creatable = True
Attribute VB_PredeclaredId = False
Attribute VB_Exposed = False
Option Explicit

' 2/8/01 JF
' 3/8/01 JF -   Block Read Added - watch for Block > File Size
'

Private Type TCMN
    FileName As String
    FileSize As Long
    Delin As String
    Buffer As String
    BufferLen As Long
    BufferPos As Long
    BytesDone As Long
    EofFlag As Boolean
    Channel As Integer
End Type

Private cmn As TCMN

' ---
Private Sub Class_Initialize()
   cmn.Delin = vbCrLf
   cmn.BufferLen = 100000
End Sub

' ---
Public Function Create(FileName$) As Boolean

    cmn.FileName = FileName
    Create = False
    cmn.Buffer = ""
    cmn.Channel = 0
    cmn.EofFlag = False
    cmn.BufferPos = 1
    cmn.BytesDone = 0
    ' ---
    If FileExists(FileName$) = False Then
       MsgBox "cReadFileStream: " + FileName$ _
               + "File not Found"
       Exit Function
    End If
    ' ---
    If FileExists(FileName$) Then
       cmn.FileSize = FileLen(cmn.FileName)
       cmn.Channel = FreeFile
       Open FileName For Binary Access Read As #cmn.Channel
       Create = True
    End If
End Function

' ---
Public Function ReadDelineatedLine() As String
    Dim Q&, L&

    If cmn.Channel = 0 Then
       MsgBox "cReadFileStream - ReadLine - but file not Open"
       cmn.EofFlag = True
       Exit Function
    End If
    ' ---
    If cmn.EofFlag Then
       MsgBox "cReadFileStream - Read Past End of File"
       Exit Function
    End If
    ' ---
    If InStr(cmn.BufferPos, cmn.Buffer, cmn.Delin) = 0 Then
       Call LS_FillBuffer
       ' --- When File completely Read then append Delin if Needed
       If cmn.BytesDone = cmn.FileSize Then
          If Right$(cmn.Buffer, Len(cmn.Delin)) <> cmn.Delin Then
             cmn.Buffer = cmn.Buffer + cmn.Delin
          End If
       End If
    End If

    ' ---
    Q = InStr(cmn.BufferPos, cmn.Buffer, cmn.Delin)
    If Q Then
       L = Q - cmn.BufferPos
       ReadDelineatedLine = Mid$(cmn.Buffer, cmn.BufferPos, L)
       cmn.BufferPos = Q + Len(cmn.Delin)
    End If
    If Q = 0 Then
       MsgBox "cReadFileStream - Read - Unexpected Error" _
               + vbCrLf + "Delineator not Found"
    End If

    ' --- Was this the last Field of the Last Buffer
    If cmn.BytesDone >= cmn.FileSize Then
       If Q >= Len(cmn.Buffer) - Len(cmn.Delin) Then
          cmn.EofFlag = True
       End If
    End If
End Function

' ---
Public Sub ReadBlock(Block$)
    Dim BlockLen&, Q&

    If cmn.Channel = 0 Then
       MsgBox "cReadFileStream - ReadBlock - but file not Open"
       cmn.EofFlag = True
       Exit Sub
    End If
    ' ---
    If cmn.EofFlag Then
       MsgBox "cReadFileStream - Read Past End of File"
       Exit Sub
    End If

    ' ---
    BlockLen& = Len(Block$)

    ' --- Do we need to fill the Buffer
    If (cmn.BufferPos + BlockLen) > Len(cmn.Buffer) Then
       If BlockLen > cmn.BufferLen Then ' increase buffer size
          cmn.BufferLen = cmn.BufferPos + BlockLen
       End If
       Call LS_FillBuffer
    End If

    ' --- If insufficient Data left
    Q = Len(cmn.Buffer$) - cmn.BufferPos + 1   ' Bytes Left
    If BlockLen > Q Then
       Block$ = Space$(Q)
       BlockLen = Q
    End If

    ' --- Copy the data
    Mid$(Block$, 1, BlockLen) = Mid$(cmn.Buffer$, cmn.BufferPos,
BlockLen)
    cmn.BufferPos = cmn.BufferPos + BlockLen

    ' --- Was this the last Field of the Last Buffer
    If cmn.BytesDone >= cmn.FileSize Then
       If cmn.BufferPos > Len(cmn.Buffer$) Then
          cmn.EofFlag = True
       End If
    End If

End Sub

' ---
Public Function EofFlag() As Boolean
    EofFlag = cmn.EofFlag
End Function

' ---
Public Function Size() As Long
    Size = cmn.FileSize
End Function

' ---
Public Sub Free()
    If cmn.Channel <> 0 Then
       Close #cmn.Channel
       cmn.Channel = 0
    End If
End Sub

' ---
Private Sub LS_FillBuffer()
   Dim Hold$, Q&

   ' --- First time in cmn.Buffer = ""
   Hold$ = Mid$(cmn.Buffer, cmn.BufferPos)

   If cmn.BytesDone >= cmn.FileSize Then
      Exit Sub
   End If

   ' ---
   If Len(cmn.Buffer) < cmn.BufferLen Then
      cmn.Buffer = Space$(cmn.BufferLen)
   End If

   ' --- Reduce Buffer Size at End of File
   Q = cmn.FileSize - cmn.BytesDone
   If Q < Len(cmn.Buffer) Then
      cmn.Buffer = Space$(Q)
   End If

   ' --- Read a Chunk
   Get #cmn.Channel, cmn.BytesDone + 1, cmn.Buffer
   cmn.BytesDone = cmn.BytesDone + Len(cmn.Buffer)

   ' --- Add leftover chunk if needed
   If Len(Hold$) Then
      cmn.Buffer = Hold + cmn.Buffer
   End If
   ' ---
   cmn.BufferPos = 1

End Sub

Private Sub Class_Terminate()
   Me.Free
End Sub

'
' Support Routines
'
Function FileExists(Fle$) As Boolean
    Dim Q%
    On Error Resume Next
    Q = GetAttr(Fle$)
    If Err = 0 Then
       If (Q And vbDirectory) = 0 Then
          FileExists = True
       End If
    End If
    Err.Clear
End Function

9 Mar 2007 2:25 AM

Mike

Hi,

I have been away, sorry for the lateness of the reply, many thanks for the
class and for everybody's help.

Mike
Show quoteHide quote

"J French" <erew***@nowhere.uk> wrote in message
news:45e9330e.3001749@news.btopenworld.com...
> On Fri, 2 Mar 2007 12:52:25 -0500, "Robert Morley"
> <rmor***@magma.ca.N0.Freak1n.sparn> wrote:
>
>>It gets a bit more tricky, but you can try reading the file in chunks and
>>splitting each of the chunks. The tricky part (okay, not THAT tricky) is
>>in
>>making sure that you combine the last string in one chunk with the first
>>string in the next chunk. To my knowledge, Line Input works great on
>>"normal" text files, but if you want to step outside of that, you'll have
>>problems. Manual parsing is probably how you're going to have to do it,
>>unless maybe there's a third-party DLL that works better; I've never
>>looked
>>at any, so I wouldn't know.
>
> Here is a Class that encapsulates what Robert is suggesting.
>
> Save it as cRFStrm.cls
>
> Usage:
>    Dim oRFS As cReadFileStream
>
>    Set oRFS = New cReadFileStream
>    If oRFS.Create( "c:\test.txt" ) = False Then
>       ' bug out
>    End If
>
>    While oRFS.EofFlag = False
>              L$ = oRFS.ReadDelineatedLine
>              ....
>    Wend
>    oRFS.Free    (Optional)
>
>
>
>
> VERSION 1.0 CLASS
> BEGIN
> MultiUse = -1 'True
> END
> Attribute VB_Name = "cReadFileStream"
> Attribute VB_GlobalNameSpace = False
> Attribute VB_Creatable = True
> Attribute VB_PredeclaredId = False
> Attribute VB_Exposed = False
> Option Explicit
>
> ' 2/8/01 JF
> ' 3/8/01 JF -   Block Read Added - watch for Block > File Size
> '
>
> Private Type TCMN
>    FileName As String
>    FileSize As Long
>    Delin As String
>    Buffer As String
>    BufferLen As Long
>    BufferPos As Long
>    BytesDone As Long
>    EofFlag As Boolean
>    Channel As Integer
> End Type
>
> Private cmn As TCMN
>
> ' ---
> Private Sub Class_Initialize()
>   cmn.Delin = vbCrLf
>   cmn.BufferLen = 100000
> End Sub
>
> ' ---
> Public Function Create(FileName$) As Boolean
>
>    cmn.FileName = FileName
>    Create = False
>    cmn.Buffer = ""
>    cmn.Channel = 0
>    cmn.EofFlag = False
>    cmn.BufferPos = 1
>    cmn.BytesDone = 0
>    ' ---
>    If FileExists(FileName$) = False Then
>       MsgBox "cReadFileStream: " + FileName$ _
>               + "File not Found"
>       Exit Function
>    End If
>    ' ---
>    If FileExists(FileName$) Then
>       cmn.FileSize = FileLen(cmn.FileName)
>       cmn.Channel = FreeFile
>       Open FileName For Binary Access Read As #cmn.Channel
>       Create = True
>    End If
> End Function
>
> ' ---
> Public Function ReadDelineatedLine() As String
>    Dim Q&, L&
>
>    If cmn.Channel = 0 Then
>       MsgBox "cReadFileStream - ReadLine - but file not Open"
>       cmn.EofFlag = True
>       Exit Function
>    End If
>    ' ---
>    If cmn.EofFlag Then
>       MsgBox "cReadFileStream - Read Past End of File"
>       Exit Function
>    End If
>    ' ---
>    If InStr(cmn.BufferPos, cmn.Buffer, cmn.Delin) = 0 Then
>       Call LS_FillBuffer
>       ' --- When File completely Read then append Delin if Needed
>       If cmn.BytesDone = cmn.FileSize Then
>          If Right$(cmn.Buffer, Len(cmn.Delin)) <> cmn.Delin Then
>             cmn.Buffer = cmn.Buffer + cmn.Delin
>          End If
>       End If
>    End If
>
>    ' ---
>    Q = InStr(cmn.BufferPos, cmn.Buffer, cmn.Delin)
>    If Q Then
>       L = Q - cmn.BufferPos
>       ReadDelineatedLine = Mid$(cmn.Buffer, cmn.BufferPos, L)
>       cmn.BufferPos = Q + Len(cmn.Delin)
>    End If
>    If Q = 0 Then
>       MsgBox "cReadFileStream - Read - Unexpected Error" _
>               + vbCrLf + "Delineator not Found"
>    End If
>
>    ' --- Was this the last Field of the Last Buffer
>    If cmn.BytesDone >= cmn.FileSize Then
>       If Q >= Len(cmn.Buffer) - Len(cmn.Delin) Then
>          cmn.EofFlag = True
>       End If
>    End If
> End Function
>
> ' ---
> Public Sub ReadBlock(Block$)
>    Dim BlockLen&, Q&
>
>    If cmn.Channel = 0 Then
>       MsgBox "cReadFileStream - ReadBlock - but file not Open"
>       cmn.EofFlag = True
>       Exit Sub
>    End If
>    ' ---
>    If cmn.EofFlag Then
>       MsgBox "cReadFileStream - Read Past End of File"
>       Exit Sub
>    End If
>
>    ' ---
>    BlockLen& = Len(Block$)
>
>    ' --- Do we need to fill the Buffer
>    If (cmn.BufferPos + BlockLen) > Len(cmn.Buffer) Then
>       If BlockLen > cmn.BufferLen Then ' increase buffer size
>          cmn.BufferLen = cmn.BufferPos + BlockLen
>       End If
>       Call LS_FillBuffer
>    End If
>
>    ' --- If insufficient Data left
>    Q = Len(cmn.Buffer$) - cmn.BufferPos + 1   ' Bytes Left
>    If BlockLen > Q Then
>       Block$ = Space$(Q)
>       BlockLen = Q
>    End If
>
>    ' --- Copy the data
>    Mid$(Block$, 1, BlockLen) = Mid$(cmn.Buffer$, cmn.BufferPos,
> BlockLen)
>    cmn.BufferPos = cmn.BufferPos + BlockLen
>
>    ' --- Was this the last Field of the Last Buffer
>    If cmn.BytesDone >= cmn.FileSize Then
>       If cmn.BufferPos > Len(cmn.Buffer$) Then
>          cmn.EofFlag = True
>       End If
>    End If
>
> End Sub
>
>
> ' ---
> Public Function EofFlag() As Boolean
>    EofFlag = cmn.EofFlag
> End Function
>
> ' ---
> Public Function Size() As Long
>    Size = cmn.FileSize
> End Function
>
> ' ---
> Public Sub Free()
>    If cmn.Channel <> 0 Then
>       Close #cmn.Channel
>       cmn.Channel = 0
>    End If
> End Sub
>
> ' ---
> Private Sub LS_FillBuffer()
>   Dim Hold$, Q&
>
>   ' --- First time in cmn.Buffer = ""
>   Hold$ = Mid$(cmn.Buffer, cmn.BufferPos)
>
>   If cmn.BytesDone >= cmn.FileSize Then
>      Exit Sub
>   End If
>
>   ' ---
>   If Len(cmn.Buffer) < cmn.BufferLen Then
>      cmn.Buffer = Space$(cmn.BufferLen)
>   End If
>
>   ' --- Reduce Buffer Size at End of File
>   Q = cmn.FileSize - cmn.BytesDone
>   If Q < Len(cmn.Buffer) Then
>      cmn.Buffer = Space$(Q)
>   End If
>
>   ' --- Read a Chunk
>   Get #cmn.Channel, cmn.BytesDone + 1, cmn.Buffer
>   cmn.BytesDone = cmn.BytesDone + Len(cmn.Buffer)
>
>   ' --- Add leftover chunk if needed
>   If Len(Hold$) Then
>      cmn.Buffer = Hold + cmn.Buffer
>   End If
>   ' ---
>   cmn.BufferPos = 1
>
> End Sub
>
> Private Sub Class_Terminate()
>   Me.Free
> End Sub
>
> '
> ' Support Routines
> '
> Function FileExists(Fle$) As Boolean
>    Dim Q%
>    On Error Resume Next
>    Q = GetAttr(Fle$)
>    If Err = 0 Then
>       If (Q And vbDirectory) = 0 Then
>          FileExists = True
>       End If
>    End If
>    Err.Clear
> End Function
>
>

9 Mar 2007 8:41 AM

J French

On Fri, 09 Mar 2007 02:25:11 GMT, "Mike"
<m***@n-o--s-p-a-m.luusac.co.uk> wrote:

>Hi,
>
>I have been away, sorry for the lateness of the reply, many thanks for the
>class and for everybody's help.

No problem.

I recommend that you add the Delimiter as a property.

3 Mar 2007 9:51 AM

Bob O`Bob

Mike wrote:

> Hi,
>
> The files can be large and the machine that the job will be run on is itself
> ratther slow - that is why I didn't want to read the whole file and Split
> it. Thanks for the suggestion though, perhaps this is the only way to do it
> though?

define "large"...

For anywhere up to at least 40MB I would still recommend read-at-once, just
because the code is so much simpler. Performance is so good you wouldn't
see significant improvement using read-by-chunk until the file size was
at least >20MB.

Bob

--

3 Mar 2007 11:16 PM

Ralph

"Mike" <m***@n-o--s-p-a-m.luusac.co.uk> wrote in message
news:skZFh.38394$z54.12265@newsfe3-gui.ntli.net...
> Hi,
>
> The files can be large and the machine that the job will be run on is
itself
> ratther slow - that is why I didn't want to read the whole file and Split
> it. Thanks for the suggestion though, perhaps this is the only way to do
it
> though?
>
> Mike

One shouldn't 'pre-optimize'. Due to the way newer OS/VMMs handle files and
memory, many of the 'old-ways' aren't as deterministic as they once were. In
the modern OS there are likely 6 to 8 layers of indirection between your app
and the file. Even if you are reading "line-by-line" the OS is actually
buffering anywhere from 4k to 16k behind the scenes. Even when you are
loading an entire 40M file you are seldom actually getting the whole thing.
The savings you see is more related to the higher-level function and the
buffers served up to them, that to the actual file size.

The speed of a processor is less important than the amount of RAM, the
version of VMM you are using, and over-all load on the system. Try loading
the whole thing first, then revert to some high-level buffering only if
necessary. Always give the OS a chance to do its magic - he is far better at
it than you are. <g>

-ralph

4 Mar 2007 6:22 AM

J French

On Sat, 3 Mar 2007 17:16:29 -0600, "Ralph" <nt_consultin***@yahoo.com>
wrote:

<snip>

Show quoteHide quote

>One shouldn't 'pre-optimize'. Due to the way newer OS/VMMs handle files and
>memory, many of the 'old-ways' aren't as deterministic as they once were. In
>the modern OS there are likely 6 to 8 layers of indirection between your app
>and the file. Even if you are reading "line-by-line" the OS is actually
>buffering anywhere from 4k to 16k behind the scenes. Even when you are
>loading an entire 40M file you are seldom actually getting the whole thing.
>The savings you see is more related to the higher-level function and the
>buffers served up to them, that to the actual file size.

>The speed of a processor is less important than the amount of RAM, the
>version of VMM you are using, and over-all load on the system. Try loading
>the whole thing first, then revert to some high-level buffering only if
>necessary. Always give the OS a chance to do its magic - he is far better at
>it than you are. <g>

Actually, under MSDOS I got slightly burnt by assuming that files
would never have more than a certain number of records.

I concluded that the best thing was to keep as much out of RAM as
possible, using files for sorting rather than RAM, and buffered input,
interim files etc.

Things are a bit different under Win32, but I'm still mean with RAM

4 Mar 2007 9:54 AM

Bob O`Bob

J French wrote:

> Actually, under MSDOS I got slightly burnt by assuming that files
> would never have more than a certain number of records.
>
> I concluded that the best thing was to keep as much out of RAM as
> possible, using files for sorting rather than RAM, and buffered input,
> interim files etc.
>
> Things are a bit different under Win32, but I'm still mean with RAM

Things are /so/ totally different under Win32, that whatever you're
doing to be "mean with RAM" is very likely working against you.
Or, at best, just a waste of your time and effort.

Under Win32, you generally don't even get to deal with RAM in any
non-virtual way. Regardless of physical memory, each process has
several GB of VM, of which at any given time the vast majority will be
mapped to a swapfile anyway. Whatever "tricks" you may apply, and
particularly using files "rather than RAM" they're extremely unlikely
to even *equal* the performance the VMM would have given you.

Bob

--

4 Mar 2007 12:34 PM

J French

On Sun, 04 Mar 2007 01:54:47 -0800, Bob O`Bob
<filter***@yahoogroups.com> wrote:

Show quoteHide quote

>J French wrote:
>
>> Actually, under MSDOS I got slightly burnt by assuming that files
>> would never have more than a certain number of records.
>>
>> I concluded that the best thing was to keep as much out of RAM as
>> possible, using files for sorting rather than RAM, and buffered input,
>> interim files etc.
>>
>> Things are a bit different under Win32, but I'm still mean with RAM

>Things are /so/ totally different under Win32, that whatever you're
>doing to be "mean with RAM" is very likely working against you.
>Or, at best, just a waste of your time and effort.

>Under Win32, you generally don't even get to deal with RAM in any
>non-virtual way. Regardless of physical memory, each process has
>several GB of VM, of which at any given time the vast majority will be
>mapped to a swapfile anyway. Whatever "tricks" you may apply, and
>particularly using files "rather than RAM" they're extremely unlikely
>to even *equal* the performance the VMM would have given you.

I don't agree with you, I get massive performance increases from using
my own 'in program' buffering, these tail off around 100kb, it is
pretty obvious why, any disk operation involves heavy work by the
operating system, but this gain decreases by a factor of two so
increasing a buffer from 100kb to 200kb is going to halve the
/remaining/ number of disk operations.

The more RAM you use, the more the VMM is going to have to page in and
out, so the more likely it is to thrash. Sorting smallish buffers and
then merging them has to create a heck of a lot less disk activity
than reading a large file into RAM and sorting it there
- ditto for reading in a large file in one gulp and sequentially
processing it updating variables that exist before or after that gulp
was allocated.

The nearer variable memory the less likely a VMM caused disk
read/write.

Since, once a buffering system has been written, it then becomes a
'programming free' option, so one benefits from something that is
probably easier to use than whipping the entire file into memory and
you don't take any risks if a 2mb file suddenly becomes 1gb.

3 Mar 2007 12:50 AM

PeterD

On Fri, 02 Mar 2007 16:58:20 GMT, "Mike"
<m***@n-o--s-p-a-m.luusac.co.uk> wrote:

Show quoteHide quote

>Hi,
>
>I have been trying to use Line Input to read a file line by line
>
>MS e.g
>Dim TextLine
>Open CommonDialog1.FileName For Input As #1   ' Open file.
>Do While Not EOF(1)   ' Loop until end of file.
>   Line Input #1, TextLine   ' Read line into variable.
>   Debug.Print TextLine   ' Print to the Immediate window.
>Loop
>Close #1   ' Close file.
>My problem is that although these files are mostly plain text - I just
>usually use notepad to work with them they contain 'substitute' characters
>(0x1A) and line input stops when it encounters one.

0x1A is 'EOF' (end of file) so this is normal action. You may have to
read it as a binary file (then line input is a no-go... <g>)

Show quoteHide quote

>How can I read this file
>in VB6 line at a time? All I need is the plain text.I have triedDim fnum As
>Long
>Dim bytvar As Byte
>Dim getValue As String
>fnum = FreeFile()
>Open CommonDialog1.FileName For Binary As #fnum
>While Not EOF(fnum)
> Get #fnum, , bytvar
> Debug.Print Chr(bytvar)
>Wend
>Close #fnum
>
>But it is slow and I would have to search for a crlf in order to know when I
>have reached the end of the line wouldn't I?
>
>thanks
>
>Mike
>

3 Mar 2007 9:52 AM

Bob O`Bob

PeterD wrote:

>
> 0x1A is 'EOF' (end of file) so this is normal action. You may have to
> read it as a binary file (then line input is a no-go... <g>)

Indeed, if it's got any of those in it then it is by definition NOT a text file.

4 Mar 2007 6:25 AM

J French

On Sat, 03 Mar 2007 01:52:32 -0800, Bob O`Bob
<filter***@yahoogroups.com> wrote:

>PeterD wrote:
>
>>
>> 0x1A is 'EOF' (end of file) so this is normal action. You may have to
>> read it as a binary file (then line input is a no-go... <g>)

>Indeed, if it's got any of those in it then it is by definition NOT a text file.

Well ... maybe it is a slightly corrupted Text file
- say a confused Text file ...

I used to get those problems when using different text editors, some
appending Ctl Z others not, but keeping the blighters.