read_fasta_stream

read_fasta_stream is the streaming counterpart to protfasta.read_fasta(). It takes an identical set of arguments, but instead of reading the whole file and returning a fully-materialised dict or list, it returns a generator that yields one sanitized record at a time. This keeps peak memory bounded and makes it possible to process FASTA files far larger than RAM (tens or hundreds of millions of sequences).

At its simplest:

import protfasta

for header, sequence in protfasta.read_fasta_stream('huge.fasta'):
    # process each record on the fly
    if len(sequence) > 1000:
        print(header, len(sequence))

Because a lazily-produced result cannot be a dictionary, the dict-versus-list distinction of read_fasta collapses: read_fasta_stream yields (header, sequence) tuples by default, or [header, sequence] lists when return_list=True.

read_fasta vs read_fasta_stream

The two functions share the same signature and apply the same sanitization steps in the same order. Choose between them based on the size of the file and the shape of result you need:

protfasta.read_fasta() loads the file and returns every record at once as a dict or list. Use it whenever the dataset fits comfortably in memory - it is the more convenient interface and lets you index and re-iterate the result freely.

read_fasta_stream yields records one at a time and never holds more than a single record’s sequence data in memory. Use it for files that do not fit in memory, or when you only need a single pass over the records.

What differs under streaming

The full parameter set of protfasta.read_fasta() is supported, but a few arguments behave slightly differently because a stream never sees the whole file up front:

Return type. return_list=False (default) yields (header, sequence) tuples; return_list=True yields [header, sequence] lists. There is no dictionary return type.

When errors are raised. All argument validation is eager - an invalid keyword combination raises immediately, before any record is produced. Data-dependent failures, however, are inherent to streaming and are raised mid-iteration, at the offending record: a duplicate header, a duplicate record or sequence under a 'fail' action, or an invalid residue under invalid_sequence_action='fail' / 'convert'.

verbose output. Per-total summaries (for example, “removed 5 of 100 duplicate records”) can only be reported once the generator has been fully consumed, so they are emitted at exhaustion rather than up front.

output_filename. When provided, each sanitized record is teed to disk as it is yielded. The output file is therefore only complete once the generator has been fully consumed, and it must differ from the input filename.

Memory characteristics

Peak memory is O(single record) for the sequence data itself. The duplicate- and uniqueness-handling actions add auxiliary bookkeeping that grows with the number of records (headers plus 16-byte sequence digests - never whole sequences), so they are still far lighter than a full load. When expect_unique_header=False and every duplicate action is 'ignore', no bookkeeping is allocated at all and memory is flat regardless of file size.

Basic usage

import protfasta

for header, sequence in protfasta.read_fasta_stream('huge.fasta'):
    if len(sequence) > 1000:
        print(header, len(sequence))

Using a header parser

As with read_fasta, the optional header_parser argument is a callable (str) -> str applied to every raw header. This is commonly used to extract a UniProt accession from a structured header:

import protfasta

def uniprot_id(header):
    # '>sp|P12345|NAME_HUMAN ...' -> 'P12345'
    return header.split('|')[1]

for acc, seq in protfasta.read_fasta_stream('uniprot.fasta',
                                            header_parser=uniprot_id):
    ...

Sanitizing on the fly

Every sanitization option available to read_fasta works while streaming. For example, to stream a very large file while converting non-standard residues and dropping duplicate sequences:

import protfasta

stream = protfasta.read_fasta_stream('huge.fasta',
                                     invalid_sequence_action='convert',
                                     duplicate_sequence_action='remove')

for header, seq in stream:
    ...

Streaming filter + write

You can stream through an input file, keep only the records you care about, and write them out in bounded memory:

import protfasta

keep = []
for header, seq in protfasta.read_fasta_stream('huge.fasta'):
    if 100 <= len(seq) <= 500:
        keep.append([header, seq])

protfasta.write_fasta(keep, 'filtered.fasta')

Alternatively, let read_fasta_stream write the sanitized output for you as it goes via output_filename - just remember the file is only complete once the generator has been fully consumed:

import protfasta

stream = protfasta.read_fasta_stream('huge.fasta',
                                     invalid_sequence_action='convert',
                                     output_filename='clean.fasta')

# fully consume the stream so the output file is written in full
for header, seq in stream:
    ...

For usage examples see the Examples page. Full API documentation is shown below.

Documentation

protfasta.read_fasta_stream(filename: str, expect_unique_header: bool = False, header_parser: Callable[[str], str] | None = None, check_header_parser: bool = True, duplicate_sequence_action: str = 'ignore', duplicate_record_action: str = 'ignore', invalid_sequence_action: str = 'fail', alignment: bool = False, return_list: bool = False, output_filename: str | None = None, correction_dictionary: dict[str, str] | None = None, verbose: bool = False, silence_warnings: bool = False) → Iterator[tuple[str, str] | list[str]][source]

Stream a FASTA file record-by-record, sanitizing as it goes.

This is the streaming counterpart to read_fasta(). It takes the same arguments but, instead of loading the whole file and returning a dictionary or list, it returns a generator that yields one sanitized record at a time. This keeps peak memory bounded and makes it possible to process files far larger than RAM:

for header, seq in read_fasta_stream('huge.fasta'):
    ...

Warning

The duplicate/uniqueness-checking defaults differ from read_fasta(). To keep streaming flat in memory by default, this function defaults to expect_unique_header=False and duplicate_record_action='ignore', whereas read_fasta() defaults to expect_unique_header=True and duplicate_record_action='fail'. As a result, duplicate headers and duplicate records are not detected by default when streaming. This is deliberate: those checks each cost O(records) memory (a running set of headers / digests), which would defeat the point of a streamer built for files larger than RAM. Re-enable them explicitly (e.g. expect_unique_header=True) if you need them – a one-time warning will then flag the memory cost. See the Memory section below.

Because a lazily-produced result cannot be a dictionary, the dict-versus-list distinction of read_fasta() collapses: this function yields (header, sequence) tuples by default, or [header, sequence] lists when return_list is True.

Memory

By default this function is flat in memory: only one record is held at a time and no per-record bookkeeping is kept, so peak memory is independent of file size and files far larger than RAM stream fine. This is why the duplicate/uniqueness-checking defaults differ from read_fasta() (see the warning above).

The duplicate/uniqueness checks are still available, but each needs to remember what it has already seen, so enabling one adds O(records) auxiliary state (hundreds of MB or more on very large files):

expect_unique_header=True keeps a running set of every header;
duplicate_record_action in ('fail', 'remove') keeps a running header -> sequence-digest map;
duplicate_sequence_action in ('fail', 'remove') keeps a running set of sequence digests.

Whenever a memory-growing check is enabled a one-time warning is emitted (suppress it with silence_warnings=True).

All argument validation is performed eagerly, at call time, so bad keyword combinations raise immediately (before iteration begins). Data-dependent errors – a duplicate header, a duplicate record or sequence under a 'fail' action, or an invalid residue – are inherent to streaming and are therefore raised mid-iteration, at the offending record, rather than up front.

The same processing steps as read_fasta() are applied, in the same order (header uniqueness, duplicate records, duplicate sequences, invalid residues). See read_fasta() for a full description of each argument; the notes below cover only where streaming differs.

Parameters:

filename (str) – Path to the FASTA file to read.
expect_unique_header (bool, optional) – As in read_fasta(), but defaults to ``False`` here so that streaming is flat in memory by default. When True a running set of seen headers is kept (O(records) memory), and the default duplicate_record_action='ignore' is promoted to 'fail' (unique headers already preclude duplicate records), so True works on its own. Default False.
header_parser (callable or None, optional) – As in read_fasta(). Default None.
check_header_parser (bool, optional) – As in read_fasta(). Default True.
duplicate_sequence_action (str, optional) – As in read_fasta(). The 'fail' and 'remove' variants keep a running set of 16-byte sequence digests (never whole sequences), i.e. O(records) memory. Default 'ignore'.
duplicate_record_action (str, optional) – As in read_fasta(), but defaults to ``’ignore’`` here so that streaming is flat in memory by default. The 'fail' and 'remove' variants keep a running header-to-digest map (O(records) memory). Default 'ignore'.
invalid_sequence_action (str, optional) – As in read_fasta(). Every mode is a per-record decision and streams without extra state. Default 'fail'.
alignment (bool, optional) – As in read_fasta(). Default False.
return_list (bool, optional) – If True, yield [header, sequence] lists (matching the internal format of read_fasta() with return_list=True); otherwise yield (header, sequence) tuples. Default False.
output_filename (str or None, optional) – If provided, each sanitized record is written to this path as it is yielded (60 residues per line, as in write_fasta()). The file is only complete once the generator has been fully consumed. Must differ from filename.
correction_dictionary (dict or None, optional) – As in read_fasta(). Default None.
verbose (bool, optional) – If True, print an opening message and, once the generator is exhausted, a summary of removed/converted counts. Per-total summaries are only emitted at exhaustion because a stream never sees the file total up front. Default False.
silence_warnings (bool, optional) – If True, suppress the one-time warning emitted when a memory-growing duplicate/uniqueness check is enabled (see the Memory section above). Default False.

returns:

A generator yielding (header, sequence) tuples (or [header, sequence] lists when return_list is True) in file order. Sequences are upper-cased and sanitized.

rtype:

Iterator[tuple[str, str]] or Iterator[list[str]]

raises ProtfastaException:

Eagerly, for invalid argument combinations; lazily (during iteration) for data-dependent failures.