Jan Ischebeck wanted to discuss GNUe integrator, because I probably will need it soon. He felt it should be able to:

1. import data from databases
2. export data to databases
3. modify data in a rule engine
4. keep two tables in different databases in sync
5. export data to different formats (flat file, csv file, ...,)
6. load data from different formats (flat file, csv, XML ....)
7. load data from a logfile (always just load the new added part)

Stan Klein said he would like to be able to store data in XML files and to connect forms and reports to those files just as though as the data were in a database, i.e., changing the data would have the effect of editing the XML files. Jan said this would not be done by Integrator as such, but by adding an XML-file "database driver" to GNUe Common.

Stan said he would also like to be able to use Integrator to do "versioning" on a database, to load completely different sets of data, for example for mathematical "what-if" modelling. In the past, he had had to import and export as ASCII files to do this. As the schema had over 15 tables, this had been quite messy. Running the model involved creating about 35 additional views and tables that did certain joins of the design and workload data and calculated statistical parameters for queuing formulas. He also reported that he had found some python code that converts from Dbase format to an SQL equivalent, and wondered if this might be useful for Integrator. Jan agreed - We would modify it to become another dbdriver.

Elsewhere, Jan posted a sample mapping definition. It just the first idea. There is still some stuff missing (definitions about merging/spliting rows,...) and some stuff is defined twice (datasources). Nevertheless the file format should allow some standard datasource->datasource mappings without needing triggers. Jason asked why there were seperate <input> and <output> tags. Jan explained it was to differentiate between between datasources used for loading and storing data: But it seems to be better to just define datasources in a header and then just use them in the mapping. Jason also asked Why a separate <process> tag? Why not keep it part of <mapping>. Jan said that different mappings should not interact with each other, but processes in a mapping could be related. Jason asked how flexible the <merge> and <split> tags would be - Perhaps merges and splits should be handled only by triggers? Jan did not think we should require people to write trigger even for so simple actions. The very basic mapping stuff should be possible without any triggers.

On IRC, Derek Neighbors (derek) agreed that splits and such should be easy - but im not sure they shoudl be implemented as you speak or as triggers - nothing would prevent the integrator UI from doing all the work. To me the tag set you propose seems confusing (and i do lots of integration) - i have seen the corresponding 'trigger' way to do things it might be even more confusing. But as i learned from my xslt ventures doing complex things in xml SUCKS - so a simple split is probably much cleaner in XML as would be a simple mapping. However when things get complex the spec would probably be a nightmare. He could give some practical examples of how complicated integrations could get.

Later, he commented the thing that struck me was the mapping comment vs process - i can tell you from my experience RARELY can you just 'map'. Usually there is some form of conversion process in the middle. Jan Ischebeck (siesel) said I just meant that a "user" would like to define mappings instead of a process. Derek replied again i state im torn between implmentation and UI - hopefully a user never sees the 'definition' file - they see a UI to build mappings - and those mappings need to be powerful - not just one-for-one mappings, but concatanations, text slicing and splicing and so on. He felt that trying to do this in XML was sadistic, and that the answer is a combination of the two - some things will make well as xml elements to map - otehrs will not. Real-life situations tended to be fairly complex. Jason Cater (jcater) agreed - an xml mapping approach quickly degenerates into a lot of clutter outside of 1:1 or other *very* simple combines.

Jan saw the biggest problem in making trigger a MUST, that we have to parse python scripts to edit mappings with designer, giving an example. Derek would be ok with some of this in xml if the designer is writing it all out - my only demand is i can go to triggers and get dirty if i have to. Jan felt that IMHO integerator should support that mergemask, splitmask stuff and nothing more in XML. Jason liked this idea - anything more complex would be handled by a trigger. Should we make the splitmask capable of handling regular expressions? That could be powerful, but might not be intuitive for regular users. Jan gave a possible example - he couldn't remember how trigger definitions looked in forms ;) Jason said triggers contain or point to python code - not to another xml structure. Jan felt that Integrator triggers should work in the same way.

Jan Ischebeck had some points about appserver and forms and 3- and 2- tier setups. On security, GNUe needed to have different forms/reports/processes(GPD) for different users, probably by adding <if user=< code to navigators process definitions and let navigator provide the right form definition for the right user to forms, assuming that automatically generating forms out of database/appserver metadata was not possible.

Stan Klein's view was that security in GNUe has to come mainly from the operating system and database maangement system. GNUe's role was to provide features that facilitate the use of the operating system and database capabilities. Jan felt that IHMO the main job of the operating system is to provide a secure environment, authentification can be done by the operating system, but it doesn't have to. The security provided by authentification done just in the operating system is not necessarily more secure as when authentification is done by something else. Authentication could still be spoofed in either direction.

Stan felt that Jan's suggested approach for Navigator to provide the right definition file for the right user would be easier if definition files - such as .gpd (GNUe Process Definitions), .gfd (GNUe Forms Definitions) and .grd (GNUe Report Definitions) - had a basic "includes" functionality. He felt that if this can be done without being vulnerable to bypassing, it is probably the easiest way to get a version of Role Based Access Control. Reinhard Müller felt this would become unmaintainable on large systems. Stan disagreed - with developments like Security Enhanced Linux, I don't see how maintaining such a system is any more difficult than maintaining an equivalent system implemented a different way, such as internally within appserver.

Alternatively, Stan suggested I think there are some database systems that provide access control by field. If such a database system is available, I would recommend using it with GNUe. AppServer. as the interface between the database and the Forms client, would need to be aware of this. Reinhard felt this could work, but would mean AppServer using a seperate login for each Forms user - he had been working on AppServer having just one login to the database server. He added If we want to remain portable and database independent, we don't have another choice IMHO than doing access control within Appserver. Whether this is secure or not only depends on the quality of our own code. Stan agreed that GNUe's own security would provide the minimum, but would like to give users the option of increasing security and implementing particular enterprise security policies by configuring the system to make the operating system or database responsible for providing the authentication, access control, and security logging. He gave some examples of how this would work. The "hooks" for this probably belonged in the existing database adapters, plus a new set of operating system adapters, in Common. Derek Neighbors felt strongly that Security/Authentication should be happening inside of common as The same security must work the same for all products and it can NOT require appserver. He noted There are some security adapter plugins and documentation for them in CVS. They are not the end all of what security should be, but rather a practical starting implemenation. Stan emphasised that he was not talking about *requiring* the use of anything, but different systems in different organisations needed different levels of security. Low level security would indeed be acheivable just by using the security within GNUe. However, to achieve the higher levels of security needed in other cases, then operating system security and database security were also needed, as part of the overall secuity strategy.

Earlier, Jan also thought we need a delegates/plugin architecture for common to get a common which can be very powerfull and very lean at the same time. This would enable developers of two-tier applications (i.e. Forms talks direct to database) to access functionality like authentification adapters without using the AppServer. Jason Cater said That's pretty much how everything in GNUe-Common works and why we keep pushing the AppServer guys to reuse Common as much as possible.

Stan made some general comments about authentication systems, which were usually based on either something the user knows (e.g. password or an encryption key) or something the user has (e.g. biometric checks). There are a wide variety of means for attacking authentication systems, including sniffing passwords sent across the network, social engineering (e.g. tricking a legitimate user into revealing a password to a malicious intruder), and password guessing (possibly testing entire dictionaries of words). However, the most serious potential attacks would be those that exploit programming errors and other vulnerabiilties to bypass the authentication system and gain access to the platform and database without logging in. For example, an attacker might try to crash AppServer to get to a Python traceback and Python prompt, or to a system prompt, giving them a foothold for further entry. That's why I think that GNUe should base as much of its security as possible on the operating system. Jan said he was not sure how popular high security operating systems like NSA Linux, or the Linux Security Modules will become in the next years, which was why I would like that gnue gets a very flexible security structure. A structure which let the user choose about using the very secure version of GNUe possibly bundled with NSA Linux, or a less secure one with some more features :). Stan agreed about flexibility - he might appear focused on the high security requirements but that was because they are the greatest challenge. It is easy to back off and provide less security. All a user has to do is select Windows as the operating system, and most security goes away. :-) It was wrong to assume that all Small or Medium Enterprises would only need low security - a small law firm or accounting firm might have very stringent security requirements arising from the kinds of legal or accounting matters they handle.

Elsewhere, Robert Jenkins said Security and access control are very important, but please don't go to impractical levels. The most vulnerable part of the system is the database itself - unless you are going to encrypt all stored data, there is nothing to stop someone with either physical or program access to the database server from reading what they like. To determine what parts of the application users could see and use, All the business apps I've worked with have been based on either assigning users and forms/fields a numeric level, & only visible / editable to the user if that user's security level is >= to the form or field level or a similar approach but using User-group type Bitmasks - each user has (e.g.) a 32 bit security mask which enables viewing or editing for forms / fields with matching bits. He agreed with Reinhard that Having multiple forms for different levels is just not practical for a serious application.

He also said Presumably the usernames & passwords will be stored in the main database, so the program must have a built-in or configured 'fixed' password to be able to verify user logins. Neil Tiffin felt This sounds good for phase I, but having user passwords in the database will be suboptimal in any situation that has more than a few users. From a maintenance standpoint we should be able to use LDAP or Active Directory to validate passwords and not store them in the database. Reinhard said We were talking about using PAM for authentication, which would mean (from my understanding) that we can use at least LDAP as well as simple shadowpasswords and more. IMHO there are a _lot_ of thing that make sense to generally store in the database, but not the passwords. Neil found some references to using PAM with python on the web, but nothing more recent than 1999.

John Lenton (Chipaca) asked is there any way to do a limited select? i.e. select foo from bar where baz limit 5. Jason Cater (jcater) said not. John was wanting to do a lazy load of some stuff, but to bring _something_ on startup. James Thompson (jamest) said you do that anyway in our system - the default is 10 records - set the datasource to prequery. Jason expanded our system only loads records as it needs them. James said that, to change the default, you just set the cache="" attribute of the datasource tag. This interacted with the rows displayed on a form, so that if you have a entry on screen that has rows set to 15, it would load two lots of 13 records and display the first 15.

Derek Neighbors (revDeke) noted that he had developed some GNUe Forms for use with DCL for people who did not want to use the normal web-based front-end - note they are pretty ugly as they are created with a wizard in Designer, but they had required almost no effort to create. He noted that the same GNUe Form Definition (.gfd) files could soon be used on a curses interface - short of that i think we plan on making some 'email' clients - so that you can update things back and forth better via email - /me isnt really all that sold on it, but its do able - and the fsf would like such a thing. He did not see the justification for an emacs mode for somethign that is not emacs like at ALL - an editor shouldnt be a project management application :) With current functionality, you could set up DCL to accept tickets via an e-mail gateway from any mail client, including emacs. He felt you might like request tracker better - i think its seriously lacking in features compared to dcl - but to some people emacs mode is more important than actually doing productive work :( He was not bagging on emacs... you can ask jcater its the first thing i install on a new machine - but emacs really is crappy for many things - the things its good at, its REALLY good at. Discussion drifted to a general text editor debate (emacs or vi, with kate mentioned as a possible third way).

Ryan Fox (larsu) was the original developer of Nola. i saw some mailing list posts about what you're trying to do with acclite, and wanted to offer my assistance with code, or just giving you insight as to what I was thinking :) Peter Sullivan (psu) said that GNUe's involvement with Nola had started as just patching it, but they had ended up forking it as acclite, and were now looking at using it as the financials part of GNUe Small Business.

Ryan was no longer with Noguska, but though it was unlikely they would have accepted the patches anyway - the latest i heard was they were doing a complete rewrite of nola anyhow. my guess is so they aren't bound by the gpl/other people's copyrights on the new version. Peter said it was generally accepted that submitting patches implied transferring the copyright to the owner of the patched code, but that there were some legal concerns about this. This was part of the reason GNUe asked all people submitting code to do a formal copyright assignment to the Free Software Foundation (FSF) to avoid problems later. Jason Cater (jcater) clarified that the Copyright Assignment, iirc, does not take any copyright rights away from the person writing the code - but it extends the rights to the FSF as well - so in essence, you are granting them the right to defend your code.

Later, Derek Neighbors (derek) said he had talked for sometime to a lady at noguska about the gpl and submitting patches to them - probably about the time you stopped working on nola. He explained long term we felt that we were better doing a fork - as it seemed as though it would be hard to get changes to happen going that route - and since we were not tied to php frontend much likely it would be cumbersome to maintain that. He was now doing gnue-sb (gnue small business) which will be a gnue framework approach to the similar domain. He noted the other issue i had with nola was some dependencies/leaning towards print shop. Ryan said he did not know anything about GNUe forms, but intend to learn. Derek asked whether Ryan was looking to take nola and mold it from where it sits etc - is web frontend high priority to you etc? Ryan said i don't have any vision for it at the moment, except i'd like to see it integrated to the GNUe way of doing things.

Jason wondered why Nola had been written in php + mysql in the first place. Ryan said MySQL had been chosen because a requirement was to run server cross platform, and be $$ free. Derek asked did anyone (clients) ever use nola? we tried to call noguska and ask for references that were using it at got nowhere - it seemed they considered it doa - and were using non free non web for most clients. Ryan said it had never been a priority product for Noguska - originally, they developed accounting (and print shop mgmt) in qbasic, dos based and Nola was intended to provide an upgrade path for this. He was not aware of it being heavily tested in production anywhere.

Derek said he had installed and configured Nola and messed with it, but had now started to go more from scratch witth gnue-sb - as it was easier than dissecting what was there (and i had no intentions to maintain the php web part) - so as soon as we decided we were not going to try to be nola compatiable upstream i started view it as a 'reference' instead of a bsae - though for accounts receivable and payable and general ledger i might change my mind. Ryan asked what are your thoughts on the schema? do you think that'll stay mostly intact, or vastly different? Jason, as the person doing the initial research, said nola had one of the better schemas - we looked at sql-ledger as people raved about it but thought that Nola's looked more mature.

It was asked about looking at commercial applications' schemas, but Jason said no way we can do that - no way at all as schemas were copyright, and strongly protected by most commercial application vendors. For example, his own organisation used Great Plains, but he will not even glimpse at it - as that's a conflict of interest big time. In any case, he thinks it's dangerous to just try to emulate a commercial package's schema and fields - even if you avoid any legal trouble - free software should aim to provide better functionality than similar proprietary systems. For example, he recognised that Oracle SQL*Forms had influenced the initial design of GNUe Forms, but we are not, however, trying to do a duplicate - we do things differently than they do - some things very differently. it's like one's cooking style - if you grow up in a Southern kitchen and you start up your own restaurant with your own home-made original recipes - odds are anyone eating at your restaurant can tell you were influenced by southern styles :) - that's my analogy for the day.

James Thompson (jamest) said he had heard very little about bugs with the 0.4.1 release, but i guess we could wrap up a 0.4.2 release. Andrew Mitchell (ajmitch) asked why roll 0.4.2 when there's no changes? James said there are changes, just not alot of them. Jason Cater (jcater) said curses didn't install - someone complained about that. James said that he and Jason had done sundry bug-fixes, and he had merged in some papo stuff too as discussed in . Jason said curses mostly works now in 0.4.2 - I would like to finish it up - but I honestly don't see me having time to do that anytime soon. What was broken was just really odd things - buttons don't fire, menus don't exit once they've fired an event. Right now the curses thing is good for demoing - but not for using. James asked are these issues in cursing or issues in uidriver or unknown?

Jason Cater announced that the forms GFD (GNUe Forms Definition) specification is undergoing some changes. The main goal of James and I is to separate layout from logic. This would mean that all old forms would stop working. We do not take changing the GFD format very lightly. However, these are changes we have wanted to make for quite some time now and sooner was better than later the closer we get to Forms 1.0. To ease the transistion, There is a utility in gnue/forms/utils called gfd04to05.py. He listed the changes to the format, and gave an example of a simple form in both the old and new formats. He also noted that the new format had hooks for layout management. Don't expect other layout managers any time soon. However, we wanted to break our format only once and be done. Also, for the foreseeable future, Designer will only support the existing character-based format, even after a few layout manager plugins are written. He also noted that the convertor script would try to convert trigger code as well, but it can only do so much, giving examples of the sorts of cases it could not handle. All of these changes have been made in cvs head for the Forms product. However, Designer is still broken and will not read the new format. We are currently working to correct this.

Some days later, on IRC, a bug was reported running a GNUe Form Definition (.gfd) with the CVS head version of GNUe Forms. James Thompson (jamest) pointed out cvs head is way different gfd format which had been introduced for what would become version 0.5.0 of Forms - so i imagine that's the issue. There is a converter in forms/utils - or we have branches of the 0.4.x stuff that has most the curses updates. The documentation had not yet been changed to reflect the new format.

Jeff Bailey (jbailey) asked Anyone around who knows about making listboxes in designed that pull their values from other tables? The designer wizard just confused me. =) He wanted a drop box list of my provinces, as defined in a tblProvinces table, to populate a foreign key field in a tblFoo. Derek Neighbors (derek) said you need to make a datasource for both tblFoo and tblProvinces - i.e. 2 separate datasources. Jeff said It looks like the wizard did that for me. Derek said then put an entry on your form that belongs to tblFoo and is for field province_id - then go to property inspector for that entry - there would be three properties that needed to be filled in:

1. fk_datasource - the name of field in tblFoo that needed populating
2. fk_key - the name of the field in tblProvince that was the primary key of that table
3. fk_description - the name of the description field in the tblProvince table