WordPress.org

Ready to get started?Download WordPress

Forums

Anyone want yet-another Amazon system? (13 posts)

  1. Matt Mullenweg
    Troublemaker
    Posted 10 years ago #

    Why don't you take a look at denyerec's code and combine your efforts? It would be better for the community and each of you.

  2. davidchait
    Member
    Posted 10 years ago #

    I don't know -- there's more than one Amazon solution for MT, right? ;)
    But really, I think we have two different approaches/solutions. Mine is about leveraging live/dynamic content, full amazon data lookups, with no database access whatsoever (I'm on a no-SQL-if-not-needed rant :) ). His is about having a simple 'gui' interface to plug in items of interest through the admin system, and archiving more 'static' data in the database.
    Or, at least that's my read just from what he's posted. They're not mutually exclusive, but each has pros and cons depending on the user and usage. His is probably easier for the mass audience, mine may be better for the power users or people more focused on content-driven 'sales'.
    Each to their own. Oh, the other advantage is that while mine does integrate into WP, it doesn't require it. You can however more tightly link the two systems with calls to allow tracking of categories and link items based off of categories seen, or as I mentioned I have another hack+'core-mod' that allows embedded float items within posts.
    We'll see. Maybe we'll find some common ground between the two.
    -d
    http://www.chait.net

  3. davidchait
    Member
    Posted 10 years ago #

    After reading up more on some of Amazon's rules for managing data, I've added some enhancements to the caching mechanisms...
    First, I made it so that so long as the cache file already exists, it won't try to recache if you've already pinged amazon in the last second. For heavyweight-usage sites, this will break down, and I'll need to find a better way to manage -- unfortunately, might require working through the database (ugh). I'm sure Amazon doesn't mind a site that once a day does say four updates in the span of a second, then does nothing for another day, but seemed like a good feature to add.
    Second, I made it so that the minimum cache refresh is 60s, and the maximum is 24h. This is so 'user error' doesn't lead one down a bad path..
    Third, I've added an option to print the cache file time at the end of a set of links, as Amazon has certain rules on the storage (and printing) of price information, which this tries to address.
    Amazon has some weird rules when it comes to storage of data. Other than pricing or availability, you can stash most data for up to 3 months. However, the moment pricing or availability data is displayed, they shift their time structuring, which makes no sense, to require updates once per hour, or once per week if you put a timestamp and a disclaimer. (Currently I have the disclaimer external to this plug-in, but I'll likely embed it in a callable function so people can add it in as desired...)
    Anyway, just thought I'd keep people apprised of the progress.
    If you haven't bounced around my site (www.chait.net) to see some of the different aspects of my system working live, please do! I'd love feedback from people who want this type of live Amazon functionality in a concise plugin.
    -d

  4. davidchait
    Member
    Posted 10 years ago #

    Anyone out there interested in testing my cg-amazon library out? I've probably got at least another few days (assuming I get time -- otherwise it could be a week or two easy!) before it'll be near 'alpha', but then maybe getting a few active testers on it will help me to get it in shape... ;)
    Obviously I need to work on docs and samples, but I also need to make sure I don't have any weird dependencies on specific functions or subsystems of the CHAITGEAR site code... :)
    -d
    http://www.chait.net

  5. davidchait
    Member
    Posted 10 years ago #

    Just another update... ;)
    I have the major code extracted, minimizing main CG code dependencies... might have one or two minor ones left. I'm 'testing' my own code by trying to keep updating it live internally as I go through the cleanup, make sure it all still works.
    I've started rewording the function names for consistency. I need to add quickie comment-docs for each function as well.
    Threw together a quickie disclaimer/license. If anyone has a good one around, let me know. :)
    Still working on the ideal approaches to caching AND meeting Amazon's desired update frequencies. Not perfect yet, but getting there... I may need to use a more accurate timing function (I need sub-second, floating-point accuracy, and I'm not sure if the php timing gives that by default...).
    And still looking for some testers! ;)
    -d
    http://www.chait.net

  6. davidchait
    Member
    Posted 10 years ago #

    Well, another ten days, time for another update.
    The last week has been hellish. Sometime over the weekend, Amazon screwed up royally by pushing out live some revisions to Amazon Web Services... which were broken to say the least. They still don't have everything back working properly, not the least of which is the method I used for running my queries.
    Long story short, I wasted a few days, hacking around with alternate solutions (rather than writing content for my site! bad...), and finally tonight I found one small piece of unrelated sample code that helped me out.
    I now am accessing Amazon's data through a different query interface. It's supposedly 2-3x faster generally, and with all the overhead I removed (I have my own helper functions rather than the monster that was there...), that's a small estimate.
    I'm looking into alternate organization & caching methodologies, for users with big databases of items they want to keep. I might break down and add a flag for whether to cache to disk or cache to the database, and restructure the system to support both AND do both equally well. You'll take the time hit on pulling either a big file or multiple small files off disk versus the time to run multiple queries across to MySQL. If you're like my hosting provider (and like most), the sql server is a different box on the network, so you take an extra hit to run the query.
    Well, enough about me. How about all you folks out there? Is there anyone at all still looking for different solutions, features, etc.?
    -d
    http://www.chait.net

  7. davidchait
    Member
    Posted 10 years ago #

    Further update (I love hearing myself talk! ;) ).
    Further revamped the query system. Now pretty cleaned up, should support all the old query types within reason.
    XML library has been cleaned up (pulled from another source, but it was missing some minor things...), and is now the basis of both my cg-amazon plugin and a complete overhaul/revision of the rss feed reader plugin (now called feedread), which is now much cleaner and more extensible to pulling in further data fields quite easily. (I should start a sep discussion for feedread, as seems many people want a small/fast feed aggregator...).
    Moved cache checking function into the cg-amazon lib instead of the user-specific file, so you just call one function and it goes and ensures ALL cached data gets updated reasonably (and more actively, rather than passively).
    Still exploring system for mapping cache datafiles on webserver to db entries, for users who want that. At the least, the ability to manage the master list of amazon entries in the database, have it locally cache/store the name of the ASIN you've entered (for). Might still store per-ASIN data on the webserver, in one or more cache files, to avoid per-ASIN lookups to the server.
    The database becomes an issue/bottleneck for a site like mine that wants to display multiple items, multiple categories, singular items, all on a single page update. Since the page is 'live', we don't know ahead of time what data might be needed, and the 'heavy' amazon data feed can be HUGE, thus you wouldn't want to grab data for ALL amazon entries at once. Thus, more likely we'd want to cache the actual amazon-'feed' data in cachefiles on the web server, fast and close to PHP, but possibly manage the master-list of products (really an ASIN list and associated user-categorization) via a single table (and one fast get-everything query) in the database.
    -d

  8. davidchait
    Member
    Posted 10 years ago #

    I feel narcissistic here... :)
    So, the new and improved CGAmazon is now up running live on http://www.chait.net.
    - Has the new XML parser interface, that sits under the new REST-based request layer.
    - Has a new caching mechanism, caches individually for singular ASINs.
    - Has an integrated wp-admin interface page for controlling the master list of Asins and categories (and basic data).
    - (Has the future ability to actually cache everything to the database, instead of to disk... that'll be slow for more than 2-3 products on a page...)
    - Has much streamlined/simplified methods for displaying random entries from a given user-defined category.
    It's making a lot of progress. I don't know how much I like the admin interface yet -- editing Asins directly in PHP was much faster, though this makes it easier to do remote. Also hard to search/sort through a large database at the moment (there's no search/sort features, aside from auto-sorting by category, and paged display set at 16 per page for now...).
    It is nice to be able to add an asin and immediately see the item details, to know you have the right one. I need to do more work though... the interface is rough, just barely started.
    -d

  9. davidchait
    Member
    Posted 10 years ago #

    In case anyone wanted to see a sample of the webmin:
    http://www.chait.net/images/cga-admin.jpg
    I moved the per-product custom notes over into the database, and added my inline-tag system to it as well so that I can inline permalinks into the product notes (that's how I get the "See Our Review" stuff for a given product -- check the homepage or reviews page...).
    The code that is managing the showing of stuff has gotten down to minimal calls, or is internal to the cgamazon lib or one of my other helper libs (xmlparser, amazonrest, inlines, sideblock).
    -d

  10. davidchait
    Member
    Posted 10 years ago #

    Now for the bi-weekly CGAmazon update! ;)
    check out http://www.chait.net/cgaindex.php to see the latest addition to the CGA feature set! the CGA database now has an (optional) index of it's own, so people can browse the amazon list like they are browsing the blog.
    Currently based off a table for the main list, with the column-titles as clickable 'sort-by' links -- this is all inherited from the web-admin interface to some extent. Clicking a given product takes you to a product details page, with price, availability, feature list, and then a breakdown of the first three customer reviews.
    Note this is not meant to be 'drop-in'. It'll want to be customized to match the index page of a site. I've been thinking about ways to direct integrate it into index.php, but it was getting a bit more hairy than I cared for, I wanted SOME differences for the amazon list pages, and I've already separated much of the substructure of my index.php out so it can be called (or not) by a referring page structure.
    I also added the functionality so that when you look at a product, IF it has 'accessories', it will query for those and list them in the sidebar code processing. If it doesn't, it'll use the 'similar' items list to query and list those instead. (I also now can grab the similar items direct from the 'primary' item data, rather than having to run yet-another query to Amazon for Similar items...)
    Anyway, that's all for the update today. Check out http://www.chait.net, the amazon product sidebars, the amazon inlines in reviews and news posts that tag amazon products, and the new product index and detail system.
    -d

  11. rayne
    Member
    Posted 10 years ago #

    Could I use this to display a list of books I've read, grouped by category (ie: Read in 2004, Read in 2003, etc...)? My current site looks like this: http://wifetalks.com/books. I'd like to do something similar in WordPress, as I'm trying to convert my site from MT and I'd like it to be somewhat similar to what I've already got.

  12. davidchait
    Member
    Posted 10 years ago #

    It's 98% there. I could build a category view like you have pretty easily, and I need to update the code a bit to work better for books (rather than electronics, which has been my main focus). Let me take a crack at it tonight and see if I can pull it together.
    -d

  13. davidchait
    Member
    Posted 10 years ago #

    BTW, I did go and add the capability to add other media types (books, DVDs, etc.) in, and sort based on the amazon categorization (in a different style layout -- still need some user-configurable styling...).
    Also, based off what you showed, I added the ability to store the authors/actors/musicians lists for the appropriate mediums, and default to the manufacturer/publisher for all others. I also added the ability to tell the system to show the manufacturer/author after the product name, and also whether or not to embed the small image url to show the item picture -- looks best in a table-based layout (right justify the image, left justify the text, space between them), but works for most anything.
    And, I just went through a MAJOR rewrite of the random-list management code. It now will only retrieve n+3 (n is your random number) total cached entries off disk from the category, but will always refresh ALL entries that need refreshing from Amazon so you don't have stale data. This way, you never pull up more than a few files off disk during page-generation, significantly lightening the load. The +3 could be a configured variable if needed.
    I can also move to having the core cached data stored embedded in the database. The reason I haven't yet is:
    1. It is MUCH faster to pull cached data off disk than across network via queries.
    2. Average user/site has more web-space than db-space allocated, and 'heavy' Amazon data feeds can get large when you start talking hundreds upon hundreds of items.
    I might look into compression or other schemes (could get at least 50% compression on the cache files), but that ups the load on the processor.
    Still open to having some people test it out for me, let me know what they think. It's really easy to add in a ton of ASINs, and I'd even be willing to write an importer that reads a list from a datafile (ASINs or other search data) if that'd help.
    I should also note that I'm investigating the ability to add other data sources into the mix, such as affiliate program data feeds from subsites within commission junction or befree. TigerDirect, etc.
    -d
    http://www.chait.net
    Amazon listing at: http://www.chait.net/cgaindex.php

Topic Closed

This topic has been closed to new replies.

About this Topic

Tags

No tags yet.