Hi Nikolai! What a nice surprise to see an experiment with Doctrine ORM here. I was just looking at the code you added in the Doctrine-branch while you posted this message. I will certainly give some feedback here after I've taken a closer look. I understand it is just a first sttep, but still interesting. What I miss now and will help showing is the main point of an ORM: mapping tables to objects (entities). That means that you are also mapping the relations between the objects. Not: an issue-item having a project_id foreighn key, but an Issue-entity belonging to a Project (and the Project having an issues collection). I'll map that in Doctrine (which can be done very quickly using ORM-designer), but then I first have to take a closer look at the whole JIssues application and how it is built up, especially the relations between the entities. But I'm very interested in that now! Thank you very much for the first step!
You may blame the J!Tracker Application for transmitting this comment.

BTW did you see the DoctrineProvider piotr_cz is working on? Here is a link to a discussion he mentions it and some comment from me in some subsequent posting about how to fix his prefix-eventlistener:
https://groups.google.com/forum/#!topic/joomla-dev-framework/QXQlAS_zhdA

Another wild idea coming up, probably nonsense, but still want to share it: you are a co-mentor of one of the 8 accepted Joomla GSOC (Google Summer of Code) projects: delivering/finishing JIssues by Zehan Zhao from China. That means that this student will be paid this summer to work on JIssues. Next week, in the next Joomla Community Magazine there will be a presentation of his project. I'm mentoring another project, but could also assist to this one... and for sure I would do that if we would try to accomplish to make a complete Doctrine ORM implementation of it. I can imagine this is totally out of scope, but wild ideas keep the world moving, hence this remark. Please ignore if not applicable.
You may blame the J!Tracker Application for transmitting this comment.

Guys can you explain why we should move to Doctrine instead of JFW Database? What are the benefits?

Thanks.

P.S.
I've never worked with Doctrine before, but feel excited to know something new.

Best regards,
Dmitry

----- Reply message -----
From: "HermanPeeren" notifications@github.com
To: "joomla/jissues" jissues@noreply.github.com
Subject: [jissues] WIP Database refactoring (aka introducing Doctrine) (#342)
Date: Sat, Apr 26, 2014 21:37
BTW did you see the DoctrineProvider piotr_cz is working on? Here is a link to a discussion he mentions it and some comment from me in some subsequent posting about how to fix his prefix-eventlistener:
https://groups.google.com/forum/#!topic/joomla-dev-framework/QXQlAS_zhdA

Another wild idea coming up, probably nonsense, but still want to share it: you are a co-mentor of one of the 8 accepted Joomla GSOC (Google Summer of Code) projects: delivering/finishing JIssues by Zehan Zhao from China. That means that this student will be paid this summer to work on JIssues. Next week, in the next Joomla Community Magazine there will be a presentation of his project. I'm mentoring another project, but could also assist to this one... and for sure I would do that if we would try to accomplish to make a complete Doctrine ORM implementation of it. I can imagine this is totally out of scope, but wild ideas keep the world moving, hence this remark. Please ignore if not applicable.
You may blame the J!Tracker Application for transmitting this comment.

—
Reply to this email directly or view it on GitHub.

Hi Dmitry! I wonder: are you one and the same as Dmitrijs Rekuns on the JGSOC mentors list, the first mentor of the JIssues JGSOC project? And if so, is your name spelled wrong there or is it a matter of transliteration from Cyrillic alphabeth?

I'm completely biased concerning Doctrine, as I work with it daily now. So take a grain of salt with everything I say about it. And: I certainly didn't say we should move to Doctrine. But I personally don't want to do PHP-projects without it anymore. And was looking at JIssues allready to use it as an example of modeling with an ORM (Object Relational Mapper, i.c. Doctrine). BTW: are you guys going to J & Beyond? For I want to show some more of this there too: http://jandbeyond.org/program/sessions/next-generation-joomla.html

JFW Database is an Active Record implementation. That means, there is a strong connection between the tables in the database and the tables in your application. An ORM decouples that by putting a Mapper in between. You can now directly define a $issues within a $project object. That $issues is a property of $project and has a collection (think of: array) of $issue-entities. A $issue-entity can hold other collections etc. You get a whole graph of objects in that way and you can easily "walk the graph" and for instance do: foreach($project->getIssues() as $issue) to list all issues in a project. It gives clear code, in which the model (= all the entities and their relationships) is used in a way that is closer to the domain you are modeling than when you would think in seperate tables, foreign keys, etc. There is another nice thing with an ORM like Doctrine: you can also model inheritance. When thinking in tables and foreign keys, you only think in one-to-one, one-to-many and many-to-many relations. But a one-to-many relationship between a project and an issue is another kind of relationship than for instance between an issue in general and specific subtypes of issues, that are inherited from an issue-supertype. In the mapping of Doctrine those relations are all nicely defined. You get a model in which teh persistence (to put it to and retrieve it from the database) is not part of. You just model the entities (domain-objects) and their interaction, everything is managed in memory. If you have some coherent things (a "transaction") done in your model, then you just call flush() and swoosh... everything is put into the database. The whole interface to the entities is via Doctrine's EntityManager. You don't use sql directly, for the interaction with the database is done by Doctrine. There is a sql-like query language, dql, with which you don't query the database tables directly, but the objects (and in the background that sql is efficiently generated by Doctrine). You normally don't use that dql directly in your model, but nicely put behind methods in so called repositories (of entities). If you have ever read Eric Evans ("Domain Driven Design") you have an idea of what I mean by entities. If you haven't read that book (from 2003): do yourself a favour and plan to do that. Especially if you want to model more complex situations.

Downsides:

• you now have a setup with seperate app, one per entity. going the Doctrine way will probably overhaul this basic structure, as you model your entities AND their relationships. You'll probably get other lines along which you seperate your whole application into parts. I'm very interested in what the structure of JIssues is at the moment, especially the interaction between the different parts.

• a mapping layer in between can be a performance issue. Doctrine has solved that mostly by caching at various levels: making proxies, generating sql, metadata-information etc. Besides that the Doctrine-project is actively interested in the HHVM-project, Facebook's PHP JIT-compiler, which gives an enormous boost to execution speed. Still, I've got some projects with Doctrine in Joomla running in production for some years now, that are not well cached, running on a shared host, and still performant enough.

Hope you don't mind me typing such a long posts here... I must certainly put more online, so I can just give a link. Hope you get interested in it. Please tell me your doubts, questions, etc.

Ciao,
Herman
You may blame the J!Tracker Application for transmitting this comment.

Hi Dmitry! I wonder: are you one and the same as Dmitrijs Rekuns on the JGSOC mentors list, the first mentor of the JIssues JGSOC project? And if so, is your name spelled wrong there or is it a matter of transliteration from Cyrillic alphabeth?

Yes it's me :) Dmitrijs Rekuns is my real fullname (it's in Latvian). And Dmitry Rekun it's like more easier to read transliteration to English :) Hehe.

Thank you for the long post, it's getting clearer now. I also watched you video which helped a lot. And yes we (me and Nikolai) are going to JAB and your session is in my list :)

Wow, great! Definitely looking forward meeting you there. I will certainly add something from JIssues & Doctrine to my presentation too. For today I'm finished and tomorrow another obligation... But I'll keep you posted; can hardly wait.
You may blame the J!Tracker Application for transmitting this comment.

Hi Herman,
glad you got interested in this idea. That's why I opened this PR at such an early stage, hope to getting some feedback before moving in very wrong directions.
Yes, the most important part of ORM (of what I understand so far) isn't yet included. TBH, the first goal was to use only Doctrine's schema tool to provide the "create" and "update" operations for setting up the database tables.

What comes next, I honestly have no idea.
Changing the whole architecture, how we interact with the database, would require a lot of working hours, I guess. But I would love to see how that might go.

About GSoC: I think that would be awesome but, as I have very little knowledge about Doctrine I'm not sure how to properly mentor the student. But yes, I know the JTracker code.
More important: The student has to agree ;)
Generally I am all in for wild ideas that keep the world moving.

Too bad I haven't seen the EntityManager service provider from @piotr-cz, his seems more flexible ;)

About the separation: That's what I am actually after... but not completely sure how to achieve it. We should start making our Apps Composer installable and define dependencies....
Hope you find some time to guide us with Doctrine.

Looking forward to see you at JAB

As far as what's under the hood goes, I think we're doing good with following Joomla's MVC and Application packages (and of course the wonderful GitHub package). And I think we should stick with that at a minimum. We don't have to use Joomla code for everything IMO, and we're already doing that well (Symfony's HTTPFoundation package, Codeguy's Upload stuff, Flysystem, and a bunch of homegrown code from @elkuku and myself). I like the wild ideas if it makes things easier for those who'll be managing this application and fits into the architecture. (Cheap plug - http://opensource.com/business/14/4/open-source-frameworks-web-based-apps) I like finding the right tool for the job and focusing more on our requirements; if Doctrine's going to help with that then let's explore before it's too difficult to make such a change.

I'm interested to see where this goes personally. This application is going to be the most often referred to use case for the Joomla Framework for quite some time, I say we show how Joomla code rocks and how we don't have to live in a Joomla only world :-)

Refactoring

So I started with the low hanging fruits.

• App Text
  • Uses only one database table, no filters, no joins...
• App Documentor (new, was part of "App Support")
  • Reads text files and stores the parsed content to the database.
  • Uses only one database table, no filters, no joins...
• App User
  • currently only the "read" operations
  • TBF

A good sample would be the code here:

• https://github.com/joomla/jissues/tree/doctrine/src/App/Text

I believe, now comes the hard part converting the other Apps...

Debugging

I also added some query logging. Currently both database access methods (Joomla! and Doctrine) are being logged:

(BTW): The ugly gray bar in the middle of the screen is normally at the bottom, but it seems hard to "screenshoot"...

Questions:

• Some sort of bind() method which should map the request data to the entity object on save.
• Validation
• Pagination

Anyone? (ping @HermanPeeren )

Nikolai you are so fast that I can't keep up with you Give me some time to read Doctrine's docs

Sorry, due to a deadline I've shut down most communication and I only now see this thread after 6 days... Will try to find some time to take a closer look later today and share some thoughts.

Two different worlds

It can be confusing when mixing different paradigms. In this case: relational databases and objects in a domain model. Two different worlds.

Joomla's Table is an Active Record implementation, where the Table-object is almost the same as the table in a relational database. We are working with the database-tables directly, making queries against the database in our model. So, our model is using the relational paradigm, the world of tables, columns, foreign keys etc. In this paradigm we hardly use any behaviour beyond CRUD, because those are the only operations we can use on those database tables.

When using ORM we split that into two separate worlds: the world of tables in a relational database and the world of objects (entities) in our domain model. The mapping connects those two worlds. The advantage is, that we can easier give richer behaviour to objects, beyond CRUD, which is closer to the behaviour in the domain we are modeling. For instance: hiring or firing an employee instead of creating or deleting one. We can also more naturally model relationships between objects: a project-tracker has a collection of issues, instead of: an issue-item has a foreign key to a project.

JIssues is built in the Joomla-way. It uses Table-objects that are mostly in the relational paradigm. Now we want to replace those Joomla Tables for Doctrine Entities. That is possible, but: now we have one leg in the world of relational databases and the other leg in the world of domain objects. There are some issues with that:

• in fact it is useless to use an ORM then. It is only extra overhead, without much benefit. We must keep in mind that we do this only as an exercise, to get some more experience in using Doctrine in combination with parts of the Joomla Framework. If we really want to profit from the use of an ORM we should start modeling, without looking at database-schemas too much. Then later we can map our model to database tables. But because it would not be very productive to throw away a lot of what is already made of JIssues, we stay in no-man's-land in the middle: replacing Joomla's database-stuff like Tables and queries by similar concepts in Doctrine.

• we now start looking for ways to do the same things as we did before, but then using the new tools: CRUD, binding, querying etc. We use Doctrine tools to get Active Record behaviour done: almost like using an electrical screwdriver to hammer a nail. Probably we won't get an optimal solution.

Where to put the Entity Manager?

Doctrine's persistence is transparent. That means, that we can build a model of interacting entities without worrying about persistence. We model projects and issues and whatever objects we need. How it is persisted is not part of our domain model: we do that in a mapping layer (the ORM). Doctrine's interface to that mapping layer is the Entity Manager. Now I used the word 'domain model' several times and it is good to realise that it is not completely synonyme to the 'model' we use in MVC. In using Doctrine in Joomla until now I have always tried to keep everything working, especially view and other presentation-parts, and only replace the Joomla model and database part (including JTable, JDatabaseQuery etc). So I used Doctrine's Entity Manager in a MVC-model (and called it JooctrineModel). I have a /Model folder and in it I 've put my extension from JooctrineModel and my folder with entities (and a foldeer with repositories if I have custom DQL-queries). In my first implementation of this, some years ago, I only had the Entity Manager in the parent JooctrineModel but at the moment I'm busy working on putting some more general behaviour in that parent model.

I like it very much that I restricted the use of that Entity Manager to my MVC-model. It prevents a kind of global use of it. In Symfony the Entity Manager is used in the controllers (or in services, which are a kind of subroutines of those controllers). It gives fat controllers and lets you put application logic into them; I prefer thin (or no) controllers. In your implementation you use a Service Container to get the Entity Manager anywhere. In my opinion that is a misuse of a Service Container, which gives globals in disguise. Abracadabra, from the black hat, whatever you want, whereever you want. Handy, but I try to avoid it. This kind of use of a Service Container is an anti-pattern.

Speaking about models: in the implementation until now, you left the DefaultModel (extends AbstractTrackerDatabaseModel) in the Text App. But when using Doctrine, you don't need that Joomla-model, providing a Joomla-database, anymore. My experience with putting a model there that has an Entity Manager injected into it, is very positive.

Binding

At the moment I'm working on a general way to bind incoming data to the entities. I want to minimise CRUD-behaviour, giving my model rich behaviour as much as possible to make a better domain model. Here is an interesting blogpost from Mathias Verraes about using command-objects for that: http://verraes.net/2013/04/decoupling-symfony2-forms-from-entities/ . I'm working on a solution to define a mapper between a form and the entities, just like you map between the entities and the database tables (but then on the other end of the game). I hope to show that, or at least will do that as far as it is finished, at J & Beyond. One of my starting points for this was that I realised that whenever some input comes in that is generated by a form, we allready know what fields should be in there, because we defined that form. I want to better use that information from the form while reading the input. That, combined with a mapping to the entities, will give an easier binding of input.

Some years ago Paul de Raaij (also from Holland) wrote some articles about the use of Doctrine in Joomla. In one of them, he was experimenting with binding: http://www.paulderaaij.nl/2012/07/29/joomla-and-doctrine-experimenting-with-automated-binding/ . I like his experiments, but don't want to use that solution: not only is he using the Entity Manager in a controller (inspired by Symfony), but in this article he is also using a very dirty trick: he is using the Entity Manager inside an entity! He uses it to retrieve the metadata-information about that entity, what fields it has etc. Although I disapprove this dirty trick, I like his binding. You can do exactly the same thing with an Entity Manager in a JooctrineModel. If you don't want to wait for my long term solution of binding via an input-form-entity-mapping, you can find plenty of inspiration how you could do it in Paul's article.

BTW, about Domain Driven Design (DDD): Mathias started this interesting DDDinPHP-group in February: http://dddinphp.org . If you are new to DDD some explanation: when someone on that list refers to "the blue book" it is Evans' DDD-book, and "the red book" is Vernon's "Implementing DDD"

Doctrine at JAB

This JAB, the Saturday afternoon and evening is completely planned for contributing to Joomla (like bug squashing etc). We could do a session with some interested people, in which I'll share everything I know about DDD and ORM. We could for instance take JIssues as an example and develop a model for it, powered by Doctrine. Looks fun to me!
You may blame the J!Tracker Application for transmitting this comment.

Herman,
believe it or not: I'm going to print out your last post, take it to the beach where I'm spending my weekend, try to wrap my head around it, and post a reply when I get back to my own machine next week.

The session at JAB sounds definitely interesting and I would like to add myself as a participant

We will have Code Sprint for JIssues at JAB. Can we change it to Herman's proposal? Or may be merge with the Code Sprint?

That whole afternoon/evening is dedicated to whatever you want to do as
long as you're doing something ;-)

On Sunday, May 4, 2014, Dmitry Rekun notifications@github.com wrote:

We will have Code Sprint for JIssues at JAB. Can we change it to Herman's
proposal? Or may be merge with the Code Sprint?

—
Reply to this email directly or view it on GitHub#342 (comment)
.

Then it will be fun :)

It might be wise not to put all chances on one card: it could be that the outcome of our Doctrine-session is, that Doctrine is very nice, but not for JIssues at this moment. I don't think it will be the conclusion, but it is a possibility. If you had things in mind to do on JAB: do them anyway. So I'm in favour of merging it as an extra and not of doing some Doctrine-session instead of the planned work. I would like to do some modeling, the DDD-work, not only Doctrine. The results of such an analysis can also be useful when, for any reason, not using Doctrine as implementation.

Like @HermanPeeren, I'd say let's not integrate ORM at this point, too much would have to be refactored. For example, I found it most convenient to use Entities instead of Models, and pass the data directly to rendering engine by controller.

If the team is interested in playing with different database abstraction, I recommend DBAL.

As for the schema, I believe it should be stored within domain folder, just like the g11n files are (ie. config/doctrine/Sub.orm.yml).

Personally, as much as I like Doctrine (heck my blog site (custom FW app) was entirely Doctrine versus Joomla Database before I went and made it flat file), I honestly think this starts to stray too far from what Joomla devs & users are comfortable with. There's already a general reluctance from people to contribute because they have a hard time understanding some of the structure here.

Agree with @mbabker

It was a nice experiment, thanks for all the comments, I've learned a lot :)