How to set foloders and populate many entries with S3 download files?

[chris.paschen]

I'm in the process of trying to migrate an existing project into Fabrik.
The entries have a few file attachments. All files are stored on Amazon S3.

I've run into several bugs (or limitations) with using the fileupload with S3:

1. Creating sub-folders with placeholders in the path doesn't seem to work (it works on local file storage, but not S3)

2. The 'Allow subfolder selection' doesn't seem to function on S3.

3. No way to just 'select' an existing file on S3 (or even on local server for that matter).
It would be preferable to just 'select' a file that is on the server, rather than go through the process of uploading the file again.
If there was some other element type that could be used to 'look-up' a file on S3 (and get the valid URL) then I could easily do some code to push that into the proper field in the table. But I can't find any element that lets me 'browse' a folder for files to use (or did I miss some configuration).

I know I'm probably 'pushing the limits' of this element, but at least #1 and #2 should be functional I would think (with S3).
If I could even get #1 to work (sub-folders/placeholders) then at least I could manage.

Anyone else able to get these things to work somehow?

 

[chris.paschen]

I was able to track-down one solution to the above problems.

1. Folder names must be proper amazon folder names - or it just doesn't create the folder (no visible error message); also, you must add a trailing slash in the folder designation in the fileupload configuration

Also, the following thread helped me understand a bit better how to use the S3 integration (even though it's old, it does help explain how things are supposed to work):
http://fabrikar.com/forums/index.ph...-with-uploads-to-amazon-s3.20465/#post-111254

But still wish I could find a solution for #2 and #3 (although having #1 working removes my need for #2 - for now).

 

[cheesegrits]

Re "no visible error message". Unfortunately, for some reason the s3 API library we use just doesn't report back an error. I've tried debugging it, but it seems to be a very low level issue, with no obvious reaosn or solution. Amazon just isn't sending an error code back.

For #2, technically we could do this for buckets, I think. There is an API call to listBuckets(). Although it would require quite a lot of work.

For #3, this is something we've had on our To Do list for a long time. Basically merge the 'image' element with the upload element. Again, a lot of work.

-- hugh

 

[chris.paschen]

Hugh,
Thanks for the reply.

Re 1 (no visible error message) - I think just having some docs might help (of course, then we'd all have to actually READ the docs) - indicating that you have to watch your folder names (esp. when using placeholders)

2. For my purposes (and I think for good data organization) there really isn't a need, knowing that it does work to use placeholders in folder names/path - because you can just create another item that will allow you to 'build' the folder structure (using a radio/drop-down/field etc) that you can then use within the path - i.e. there is a work-around

3. I'd vote for this feature. The upload element seems more like an element that has just 'morphed' over the years, but wasn't built specifically for what some (at lease myself) use it for. It might just be better/easier to try re-building from scratch with features specific to file uploading/downloading (and not specifically for images). But I understand that this might not be the highest priority right now (and I've been able to find ways to manually code around most of the issues).
If there were some good docs on how to create a new fabrik element that might help encourage some community participation in projects like this. I know I've probably spent enough time manually coding work-arounds to the file-upload element that could have just been put into coding a new element to more easily handle those features.

 

[cheesegrits]

"The docs" are the wiki. Which, as it happens, being a wiki, is open for editing. Hint hint.

I did once start writing a wiki entry on writing a plugin, then realized that basically what I was saying could be boiled down to "find the plugin which does something close to what you are doing, and copy it", as most of it was refenerince examples from existing plugins. Which is what I do - I've never written a plugin "from scratch" in the ... erm ... six (seven?) years I've been working on Fabrik. There is invariably a plugin which sorta kinda does something similar(ish), so I copy it, start by changing the names, chopping out code I know isn't relevant, etc.

Re "not built specifically for what some use it for" ... well yeah. You could pretty much say that about every non-trivial plugin we have. The upload element was built for ... wait for it ... uploading, and displaying what you upoaded.

And the problem with user contributed plugins is that they are a double edged sword ... users come and go ... plugins are forever. In other words, we get stuck with maintaining it for the rest of it's / our lives. We have a discussion going (it's somewhere in the github issues list, I think) about how to implement a separate "contributed" repo, and how to run it.

But this is also why we host on github, and encourage people to Fork Me. If there's useful code changes you have made to existing code - fixes, additions, improvements - you can submit a PR, and we'll (probably) merge them, although if it involves changing huge swathes of code, we may not. We tend to prefer small incremental changes, which we can easily review without branching and extensive testing.

I'm definitely interested in adding a "select from exising files" option, and eventually being able to phase out the 'image' element. If that's something you'd feel like doing, I'd be more than happy to help out.

-- hugh

 

[chris.paschen]

I've been 'compiling' various 'sample code' items that I'm hoping to share back with the community via the wiki (although I haven't seen a place specifically for 'sample code snippets'). One resource that I love is the 'cookbook' type books that give you a short piece of code to solve some specific issue. I've pulled together a number of different things like that (like, "How to create a date range display from 2 date elements", etc. - which just uses a calc element and 2 date elements). If there is a section for that sort of thing please point me to it as I couldn't find it and I'll start sharing some of these (which are often generated by things I've found in the forum area - but it just need to be documented better with examples). [BTW ... I've written documentation for a couple other Joomla extension developers, so I'm comfortable with that task].

As for element modifications - is there any way to do a PR for just a single element (or just elements, and not the entire fabrik 'system')?
I've used github here and there for various projects, but haven't gotten too familiar with the Fabrik github (other than to update sites).
If there isn't a github for elements, that might be an option to encourage more contributions - a special 'Fabrik Elements Github'.

I think I might start doing some PR and submitting some of the display-related modifications (like making that 'date range display' a feature of an enhanced date element).

 

[cheesegrits]

When it comes to the wiki, if there isn't an obvious place to put something, feel free to create a new section. We can always re-organize it once the stuff is in there, the important thing is getting stuff in there to start with. I don't know if what you have would be appropriate for a section under Developers?

http://fabrikar.com/forums/index.php?wiki/index/#developers

Re github, unfortunately the way github works, it really isn't easy to split things out, even with trees of modules, it makes things horrible complex. So Fabrik is one big hunk 'o' junk, with everything and the kitchen sink in it.

I'm assuming your issue is you have a fork with various changes, but you only want to submit a PR for selected commits on one element? If so, there is a way to do that, although it gets a little involved. See the update the OP did on this question:

http://stackoverflow.com/questions/...with-only-the-latest-commit-in-the-master-bra

... which involves creating a clean branch of 'upstream' (us), then 'cherry-pick' the commits form your 'origin' in to it, and create a PR from that. Which can all be a bit confusing when you are new to github. Hell, it ocnfuses me, and I've been using github for years. Still don't have more than a rudimentary understanding of how the damn thing works!

Alternatively, you can do it all by hand on the github web site, by forking our joomla3 branch to a clean copy, edit the file(s) with the changes you want to do a PR for (just copy and paste from your local clone), and create a PR from that fork.

-- hugh

 

[chris.paschen]

The DEV area of the WIKI is exactly where I think my stuff should go. I'm thinking that it should be in a 'Use Case' section - with specific 'how to' instructions for various tasks. As soon as I get done with figuring out one more code issue I should be ready to start adding some 'stuff'.

Re: github - I was just wondering if all of the elements should become a github of it's own - for just elements. However, after thinking that through, it's not all that much better than what we have now. We'd just have a kitchen sink with 2 sides to it.

I think it makes more sense to me to just fork the joomla3 branch to a clean copy.
(But I think updating the WIKI is probably more useful for others - and probably more likely for me to get something done than trying to create custom elements - at least THIS week!

Thanks

 

[cheesegrits]

I know that feeling. So many things to do, so little time.

-- hugh