AGCoL

  NCBI

Download Convert files General

  Go to top


       UA BIO5        SyMAP
Home   		Download   Docs          System
Guide   		Input   Parameters          User
Guide   		Queries

NCBI supplies FASTA formatted files for genome sequence and GFF3 formatted files for the annotation, where FASTA and GFF3 files are the input to SyMAP. Though they can be loaded directly into SyMAP, it is better to convert them.

Download
Convert files
   xToSymap NCBI Options
   Scaffolds
              
General
   Load files into SyMAP
   Editing the script
   What the ConvertNCBI script does

Reasons to convert files

  1. The NCBI name for a chromosome or scaffold are long and crowd the interface. Convert uses "ChrN" instead for chromosomes only (where the 'Chr' prefix can be removed on load); when scaffolds are included it uses the prefixes 'C' for chromosome and 's' for scaffold.
  2. Only the 'protein-coding' genes are processed.
  3. Only the set of exons from the 1st mRNA are saved in the new GFF file.
  4. Gene attributes of output GFF file:
    ID from the input gene attributes.
    Name from the input gene attributes (if it is not equal ID).
    desc is the gene description; if it does not exist, then it uses the 1st mRNA product. Symbols (e.g. %3B) are replaced with the correct character.
    rnaID is equal to the first mRNA ID. Following the ID is (n), where n=the number of mRNAs for the gene.
    proteinID (optional) is equal the 1st CDS protein-id of the 1st mRNA.
  5. It has an option to produce a hard masked sequence.
If there are problems converting the NCBI file(s), then symap will have problems loading the original NCBI files. NCBI formats can be inconsistent, so this may not take everything into account; check your files with the xToSymap Summarize function. See Tested genomes and editing the script.

Download

The following instruction were updated on 21-Jan-2026.
  1. Go to NCBI.
    As shown in Fig 1: Select the Genome link under Popular Resources on right-hand side.

  2. As shown in Fig 2: Enter you genome name followed by Search.

  3. As shown in Fig 3:
    1. From the list of files, select the Assembly you want.
    2. Select Download Package from the Download drop-down. As shown, a window will popup with options.
    3. Select the RefSeq only with the Genome sequences (FASTA) and Annotation features (GFF) files.
      Note: xToSymap does NOT work with Genbank files, only RefSeq.
    4. Select Download. By default, a file called ncbi_dataset.zip will be downloaded.

Fig 1. Popular Resources
Fig 2. Search the NCBI site.


Fig 3. Select the RefSeq only with the FASTA and GFF files.

Convert files

xToSymap NCBI Options Scaffolds Go to top

The following instructions will use the Brassica oleracea (wild cabbage) as an example.

  1. If you have NCBI .fna and .gff files, do the following:
    1. Go to the symap_5/data/seq directory.
    2. Make a subdirectory for your species, e.g. mkdir cabbN
    3. Move your .fna and .gff files to this directory.

  2. If you have the ncbi_dataset.zip file as discussed above in Download, do the following.
    1. Go to the symap_5/data/seq directory.

    2. Make a subdirectory for your species (see Project directory), move ncbi_dataset.zip to it, and unzip it, e.g. 
      symap_5/data/seq> mkdir cabbN
symap_5/data/seq> cd cabbN
symap_5/data/seq/cabb> mv ~/Download/ncbi_dataset.zip .
symap_5/data/seq/cabb> unzip ncbi_dataset.zip
Archive:  ncbi_dataset.zip
  inflating: README.md
  inflating: ncbi_dataset/data/data_summary.tsv
  inflating: ncbi_dataset/data/assembly_data_report.jsonl
  inflating: ncbi_dataset/data/GCF_000695525.1/GCF_000695525.1_BOL_genomic.fna
  inflating: ncbi_dataset/data/GCF_000695525.1/genomic.gff
  inflating: ncbi_dataset/data/dataset_catalog.json
Start the xToSymap program. The program will output statistics to the terminal and a log file. The symap_5/data/seq/cabbn directory will contain the following new directories and files: 

annotation/anno.gff	annotation/gap.gff	sequence/genomic.fna
You may remove everything but annotation/ and sequence/. However, you may want to keep the original .fna and .gff files.

xToSymap NCBI Options

For the NCBI option, the FASTA file must end in ".fna" and the annotation file must end in ".gff" (the NCBI defaults). These may be zipped files ending in ".gz".
OptionDescriptionDefault
Hard Mask NCBI genome sequences are typically soft-masked, where this option changes it to hard masked Leave as soft-mask
Scaffolds1 Scaffolds will be included in the resulting FASTA and GFF files. See section Scaffolds No scaffolds
Mt/Pt Mt/Pt chromosomes will be included in the resulting FASTA and GFF fils. Only the first occurrence will be included. No Mt/Pt
Only prefix2Only sequences with the specified prefix will be processed. None
Protein-id A new attribute called proteinID= will have the value of the protein-id of the 1st CDS for the 1st mRNA of the gene. This can be searched using the viewSymap Queries. Do not include
Verbose Print extra information, see verbose log. No extra information

1You may include Scaffolds, and then limit the input on the symap Load by setting
  Minimal length in the project's Parameters.

2For situations needing Only prefix, see exceptions.

Rules: There are variations in the text associated with the FASTA ">" header lines. The rules used by this script are as follows:

  1. If Only prefix is not blank, sequences are filtered out if the seqid does not start with the prefix. Then the following rules 2-4 are applied to the non-filtered sequences:
     
  2. Chromosomes: The name starts with 'NC_' or the header line contains the word "chromosome" (this 2nd condition was added in v5.8.2).
    • The exception is that Mt/Pt chromosomes will not be output unless Mt/Pt is selected.
      Mt/Pt: header line contains the word 'mitochondrion', 'mitochondrial', 'plastid' or 'chloroplast'.
    • Chromosomes are always output unless Only prefix is set, and the prefix does not match.
    • >seqid:
      If the ">" line contains "chromosome N", where N={number, X, Y or roman numeral}, than this number is used prefixed by 'Chr' or 'C' (if scaffolds are included). For example,
            >NC_029256.1 Oryza sativa Japonica Group cultivar Nipponbare chromosome 1
         is replaced with:
            >Chr1 NC_029256.1

      Otherwise, the word following 'chromosome' is used. For example,
            >NC_027748.1 Brassica oleracea var. oleracea cultivar TO1000 chromosome C1, BOL,....
         is replaced with:
            >C1 NC_027748.1.


  3. Scaffolds: The name starts with 'NW_' or 'NT_'.
    • They will only be output if Scaffolds is selected.
    • >seqid: 'Scaf' followed by a consecutive number.

  4. Unknown: All other ">" entries are considered "unknown".
    • They will only be output if Only prefix matches.
    • >seqid: 'Unk' followed by a consecutive number.
See Summarize to help determine how to set the options for your input.

Scaffolds

By default, the Convert option creates the genomic.fna file with only the chromosomes. However, you can have it also include scaffolds by selecting Scaffolds; see log for convert summary of cabbage.

This will include all chromosomes (assigned prefix 'C') and scaffolds (assigned prefix 's') in the genomic.fna file. Beware, there can be many tiny scaffolds. If they all aligned in SyMAP, it causes the display to be very cluttered. Hence, it is best to just align the largest ones (e.g. the longest 30); merge them if possible, then try the smaller ones. You should set the following SyMAP project's Parameters:

  1. Group prefix needs to be blank as there will be no common prefix.
  2. Minimum length should be set to only load the largest scaffolds.
    Calculate the length using the xToSymap Lengths.

General

Load files into SyMAP Editing the script What the ConvertNCBI script does Go to top

Load files into SyMAP

The above scenario puts the files in the default SyMAP directories.
  1. When you start up ./symap, you will see your projects listed on the left of the panel (e.g demos).
  2. Check the projects you want to load, which will cause them to be shown on the right of the symap panel.
  3. For the project you want to load, open the Project Parameters panel to enter the appropriate values.
  4. Then select Load Project.

Editing the script

The NCBI FASTA files are not consistent in their header lines. Hence, the parsing could be incorrect. They may also not be consistent in the GFF files, but I have not found one. Anyway, if it is not parsing the FASTA file correctly, edit the program as described here.

What the ConvertNCBI script does

The script scripts/ConvertNCBI.java executes the same code as the xToSymap NCBI Convert.

FASTA: Reads the file ending in '.fna.gz' (or '.fna') and writes a new file called sequence/genomic.fna with the following changes:

  1. Sequences are output according to the option Rules.
  2. Gaps of >30,000 are written to the annotation/gap.gff file (this value can be changed in the xToSymap interface).
  3. If Hard mask is selected, all lower case bases are changed to 'N'.

GFF: Reads the file ending in 'gff.gz' (or .gff) and writes the file annotation/anno.gff. The gff3 format has 9 columns, where the first is the 'seqid', the third is the 'type' (e.g. feature 'gene'), the last column is a semicolon-delimited keyword=value attribute list. The input file is processed as follows:

  1. The type=gene with attribute gene-biotype=protein-coding are processed. The gene line is written to the anno.gff file with the following changes:
    1. The first column 'seqid' is replace with the 'ChrN' value assigned when reading the '.fna' file.
    2. A subset of the attributes are written:
      • ID and Name from the input gene attributes.
      • desc is the gene description; if this does not exist, then it is the 1st mRNA product.
      • rnaID is equal to the first mRNA ID.
      • proteinID (optional) is equal the 1st CDS protein-id for the 1st mRNA.
  2. The first type=mRNA line for a gene is written to the anno.gff file followed by its type=exon, where they each are written with the new seqid and a subset of the attributes. The mRNA ID is saved as an gene attribute (rnaID=) to indicate which exons were saved.
     
  3. If Protein-id is selected, then the attribute protein_id= from the first type=CDS of the 1st mRNA for the gene is extracted.
Go to top

Email Comments To: cas1@arizona.edu