More bugs re: Import

[JD Kromkowski]

When you import, everything is default to string type. You get a chance to modify the table (add indexes, change types). But it is pretty clear that those changes aren’t really “totally effective” unless you do a reorg. For example, if you change a field to “integer” and index it, they are not indexed as if they are integers 1,2,10,98,1000 but as if they are strings: 1, 10, 1000, 2, 98 I am see many more issues, but I’ll just stop here. And see what kind of response with this stuff.

[Michael]

On 24/04/2014 18:27, kromkowski wrote:
> When you import, everything is default to string type.

That is true. There is currently no code to detect data type during an import from a text file. In that case if you are doing it a
lot, you can use import merge to an existing definition.

> You get a chance to modify the table (add indexes, change types).

True.

> But it is pretty clear that those changes aren't really "totally effective" unless you do a reorg.

If you change data type and don't do a reorg, then SIMPOL is forced to use the data in the format that it currently finds it. That
could mess up indexes. If you change data type you really have to consider the ramifications. Worse is importing something like a
date or time, since internally they are stored as values. The same is true in Superbase. If you convert a string field containing
dates to a date field, you first have to update the database converting the date to a DAYS() value and store that in the field.
Then change the data type to date, then reorganize.

> For example, if you change a field to "integer" and index it, they are not indexed as if they are integers
>
> 1,2,10,98,1000
>
> but as if they are strings:
>
> 1, 10, 1000, 2, 98
>
> I am see many more issues, but I'll just stop here. And see what kind of response with this stuff.

If you add an index in the same step that you change the data type, without reorganizing, then this will happen. The only thing we
could do is to force a reorganize as part of the process.

Ciao, Neil

[JD Kromkowski]

It seems to be there probably is a better way to do this (but I don't know exactly what is happening – consider my email offer).

pseudo code

get first line with field names

open the modify table and let user set the field types and indexes. (it happens at the end but it should happen up front.)

save the empty table

import the data into the proper table.

JDK

[Peter Barus]

Is there a new release? I've been out of it…

On Mon, 05 May 2014 15:42:21 -0400, Neil Robinson <neil@simpol.com> wrote:

> On 24/04/2014 18:27, kromkowski wrote:
>> When you import, everything is default to string type.
>
> That is true. There is currently no code to detect data type during an
> import from a text file. In that case if you are doing it a
> lot, you can use import merge to an existing definition.
>
>> You get a chance to modify the table (add indexes, change types).
>
> True.
>
>> But it is pretty clear that those changes aren't really "totally
>> effective" unless you do a reorg.
>
> If you change data type and don't do a reorg, then SIMPOL is forced to
> use the data in the format that it currently finds it. That
> could mess up indexes. If you change data type you really have to
> consider the ramifications. Worse is importing something like a
> date or time, since internally they are stored as values. The same is
> true in Superbase. If you convert a string field containing
> dates to a date field, you first have to update the database converting
> the date to a DAYS() value and store that in the field.
> Then change the data type to date, then reorganize.
>
>> For example, if you change a field to "integer" and index it, they are
>> not indexed as if they are integers
>>
>> 1,2,10,98,1000
>>
>> but as if they are strings:
>>
>> 1, 10, 1000, 2, 98
>>
>> I am see many more issues, but I'll just stop here. And see what kind
>> of response with this stuff.
>
> If you add an index in the same step that you change the data type,
> without reorganizing, then this will happen. The only thing we
> could do is to force a reorganize as part of the process.
>
> Ciao, Neil
>

—
Peter Barus

[Michael]

On 06/05/2014 12:29, kromkowski wrote:
> It seems to be there probably is a better way to do this (but I don't know exactly what is happening – consider my email offer).
>
> pseudo code
>
> get first line with field names
>
> open the modify table and let user set the field types and indexes. (it happens at the end but it should happen up front.)
>
> save the empty table
>
> import the data into the proper table.
>
> JDK
>

Hi John,

Sorry about the late answer to this, my attention was elsewhere. I have done a considerable amount of work on the CSV converter
over the last few weeks. It is in about as good a state as I can get it without adding features like the ability to preview the
first 50 lines or so of content and then manipulate the column information. I do have that planned as a future enhancement, but it
will be part of a larger release.

Ciao, Neil

[Peter Barus]

Neil,

Libre Office does that, with the csv files, maybe they use wxwidgets or
something. Just a thought.

What are your plans vis-a-vis "apps?" Is it, or will it be, possible to
construct Android/iPad/etc-oid apps in SIMPOL?

I am switching PDAs, and amazed at the way Apple and Google are dividing
up the market for every sort of intrusive "cloud" connection. Almost
impossible to just have a smart phone or tablet that isn't plugged into
some mega-corporation's data-suckers. Before I knew how to turn it off,
all my contacts were slurped up by Google. I get the most spam from them,
from my hack-victim email friends.

Love to the family,

Peter

On Fri, 04 Jul 2014 07:28:50 -0400, Neil Robinson <neil@simpol.com> wrote:

> On 06/05/2014 12:29, kromkowski wrote:
>> It seems to be there probably is a better way to do this (but I don't
>> know exactly what is happening – consider my email offer).
>>
>> pseudo code
>>
>> get first line with field names
>>
>> open the modify table and let user set the field types and indexes. (it
>> happens at the end but it should happen up front.)
>>
>> save the empty table
>>
>> import the data into the proper table.
>>
>> JDK
>>
>
> Hi John,
>
> Sorry about the late answer to this, my attention was elsewhere. I have
> done a considerable amount of work on the CSV converter
> over the last few weeks. It is in about as good a state as I can get it
> without adding features like the ability to preview the
> first 50 lines or so of content and then manipulate the column
> information. I do have that planned as a future enhancement, but it
> will be part of a larger release.
>
> Ciao, Neil

—
Peter Barus

[Michael]

On 07/07/2014 03:28, Peter Barus wrote:
> Neil,
>
> Libre Office does that, with the csv files, maybe they use wxwidgets or
> something. Just a thought.
>
> What are your plans vis-a-vis "apps?" Is it, or will it be, possible to
> construct Android/iPad/etc-oid apps in SIMPOL?

Hi Pete,

Sorry about such a late reply, my email/news software didn't pick this up for some reason until now. The plan currently is to use
a JSON data server written in SIMPOL as a web server application and HTML5/CSS3/JavaScript for on the device applications.
Eventually we hope to generate much of that code automatically.

> I am switching PDAs, and amazed at the way Apple and Google are dividing
> up the market for every sort of intrusive "cloud" connection. Almost
> impossible to just have a smart phone or tablet that isn't plugged into
> some mega-corporation's data-suckers. Before I knew how to turn it off,
> all my contacts were slurped up by Google. I get the most spam from them,
> from my hack-victim email friends.
>
> Love to the family,

Love to you all from us.

Ciao, Neil

> Peter
>
> On Fri, 04 Jul 2014 07:28:50 -0400, Neil Robinson <neil@simpol.com> wrote:
>
>> On 06/05/2014 12:29, kromkowski wrote:
>>> It seems to be there probably is a better way to do this (but I don't
>>> know exactly what is happening – consider my email offer).
>>>
>>> pseudo code
>>>
>>> get first line with field names
>>>
>>> open the modify table and let user set the field types and indexes. (it
>>> happens at the end but it should happen up front.)
>>>
>>> save the empty table
>>>
>>> import the data into the proper table.
>>>
>>> JDK
>>>
>>
>> Hi John,
>>
>> Sorry about the late answer to this, my attention was elsewhere. I have
>> done a considerable amount of work on the CSV converter
>> over the last few weeks. It is in about as good a state as I can get it
>> without adding features like the ability to preview the
>> first 50 lines or so of content and then manipulate the column
>> information. I do have that planned as a future enhancement, but it
>> will be part of a larger release.
>>
>> Ciao, Neil
>
>

[Author]

Posts