Large External File Usage

Creating desktop or client-server database solutions?

Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller, robinmiller

Mikey
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 755
Joined: Fri Jun 27, 2008 9:00 pm

Re: Large External File Usage

Post by Mikey » Tue Feb 07, 2017 12:01 am

By the way, did you already have the indexes set on the table before you started the import?

AxWald
Posts: 578
Joined: Thu Mar 06, 2014 2:57 pm

Re: Large External File Usage

Post by AxWald » Tue Feb 07, 2017 10:50 am

Well,

this cannot be this complicated? Have you tried it the simple way?
  1. Analyze the data, and determine the data types.
    I see a VARCHAR, a DATETIME, and 12 FLOATs or DOUBLEs.
  2. Create a suitable table. Don't forget an additional ID as col 1:
    SQLite: INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,
    MySQL: INT(10) NOT NULL AUTO_INCREMENT, set as PRIMARY KEY.
  3. Split a reasonable part of your data for a test.
  4. Fire up your db manager, give it the proper information, and try an import.
  5. Read the error message, correct what you've done wrong.
  6. Once it has imported your test data, empty the table, VACUUM, and give it the full .csv.
If this doesn't work, tell us what exactly went wrong. Then we can see further ;-)

Have fun!

(PS: if you have access to MS Access, this should work too!)
All code published by me here was created with Community Editions of LC (thus is GPLv3).
If you use it in closed source projects, or for the Apple AppStore, or with XCode
you'll violate some license terms - read your relevant EULAs & Licenses!

MaxV
Posts: 1579
Joined: Tue May 28, 2013 2:20 pm
Location: Italy
Contact:

Re: Large External File Usage

Post by MaxV » Thu Feb 09, 2017 1:11 pm

Not a lot of thought wrote:I have a large file 10,000,000+ records roughly 3.5 Gigs. It is in CSV format....
I suggest you to use a:

Code: Select all

BEGIN TRANSACTION;
all you MySQL comamnds;
...
COMMIT;
It would solve your problem.
However a repeat for each line tLine in CSV would resolve too.
Livecode Wiki: http://livecode.wikia.com
My blog: https://livecode-blogger.blogspot.com
To post code use this: http://tinyurl.com/ogp6d5w

Post Reply

Return to “Databases”