Csvs-to-sqlite

Latest version: v1.3

Safety actively analyzes 629908 Python packages for vulnerabilities to keep your Python projects secure.

Scan your dependencies

Page 2 of 2

0.5

----------------
- Release 0.5.
- Foreign key extraction for mix of integer and NaN now works.

Similar issue to a8ab5248f4a - when we extracted a column that included a
mixture of both integers and NaNs things went a bit weird.
- Added test for column extraction.
- Fixed bug with accidentally hard-coded column.

0.4

----------------
- Release 0.4.
- Automatically deploy tags as PyPI releases.

https://docs.travis-ci.com/user/deployment/pypi/
- Fixed tests for Python 2.
- Ensure columns of ints + NaNs map to SQLite INTEGER.

Pandas does a good job of figuring out which SQLite column types should be
used for a DataFrame - with one exception: due to a limitation of NumPy it
treats columns containing a mixture of integers and NaN (blank values) as
being of type float64, which means they end up as REAL columns in SQLite.

http://pandas.pydata.org/pandas-docs/stable/gotchas.html#support-for-integer-na

To fix this, we now check to see if a float64 column actually consists solely
of NaN and integer-valued floats (checked using v.is_integer() in Python). If
that is the case, we over-ride the column type to be INTEGER instead.
- Use miniconda to speed up Travis CI builds (8)

Using Travis CI configuration code copied from https://github.com/EducationalTestingService/skll/blob/87b071743ba7cf0b1063c7265005d43b172b5d91/.travis.yml

Which is itself an updated version of the pattern described in http://dan-blanchard.roughdraft.io/7045057-quicker-travis-builds-that-rely-on-numpy-and-scipy-using-miniconda

I had to switch to running `pytest` directly, because `python setup.py test` was still trying to install a pandas package that involved compiling everything from scratch (which is why Travis CI builds were taking around 15 minutes).
- Don't include an `index` column - rely on SQLite rowid instead.

0.3

----------------
- Added `--extract-column` to README.

Also updated the `--help` output and added a Travis CI badge.
- Configure Travis CI.

Also made it so `python setup.py test` runs the tests.
- Mechanism for converting columns into separate tables.

Let's say you have a CSV file that looks like this:

county,precinct,office,district,party,candidate,votes
Clark,1,President,,REP,John R. Kasich,5
Clark,2,President,,REP,John R. Kasich,0
Clark,3,President,,REP,John R. Kasich,7

(Real example from https://github.com/openelections/openelections-data-sd/blob/master/2016/20160607__sd__primary__clark__precinct.csv )

You can now convert selected columns into separate lookup tables using the new
`--extract-column` option (shortname: `-c`) - for example:

csvs-to-sqlite openelections-data-*/*.csv \
-c county:County:name \
-c precinct:Precinct:name \
-c office -c district -c party -c candidate \
openelections.db

The format is as follows:

column_name:optional_table_name:optional_table_value_column_name

If you just specify the column name e.g. `-c office`, the following table will
be created:

CREATE TABLE "party" (
"id" INTEGER PRIMARY KEY,
"value" TEXT
);

If you specify all three options, e.g. `-c precinct:Precinct:name` the table
will look like this:

CREATE TABLE "Precinct" (
"id" INTEGER PRIMARY KEY,
"name" TEXT
);

The original tables will be created like this:

CREATE TABLE "ca__primary__san_francisco__precinct" (
"county" INTEGER,
"precinct" INTEGER,
"office" INTEGER,
"district" INTEGER,
"party" INTEGER,
"candidate" INTEGER,
"votes" INTEGER,
FOREIGN KEY (county) REFERENCES County(id),
FOREIGN KEY (party) REFERENCES party(id),
FOREIGN KEY (precinct) REFERENCES Precinct(id),
FOREIGN KEY (office) REFERENCES office(id),
FOREIGN KEY (candidate) REFERENCES candidate(id)
);

They will be populated with IDs that reference the new derived tables.

Closes 2
- Can now add new tables to existing database.

And the new `--replace-tables` option allows you to tell it to replace existing
tables rather than quitting with an error.

Closes 1
- Fixed compatibility with Python 3.
- Badge links to PyPI.
- Create LICENSE.
- Create README.md.
- Initial release.

Page 2 of 2

© 2024 Safety CLI Cybersecurity Inc. All Rights Reserved.