Camen Design Forum

PHP and filesystem efficiency?

append delete JJ

This is a PHP question for those smarter than I. I was debating whether to post this on a forum specifically for PHP support, but I'll post it here instead. :)

I've been trying to implement a serverside caching mechanism for my weblog system. The basic setup is currently:

1. In the constructor of the main class which is instantiated on every page, check to see if the request is a GET.
1a. Check to see if there is a cached file for this particular request URI, if there is retrieve it, serve it and then exit the script.
1b. If there isn't a cached file, set a variable notifying the final output method to cache the current URI.
2. The final method is called to output the content - before outputting the content, check to see if the content needs to be cached, if it does, cache it.

It works well, however, in order to clear the cache, I hooked a method in my database class to clear the entire cache directory if there is any SQL executed that could possibly alter a row (UPDATE, INSERT, DELETE). This is what concerns me. This means that whenever a comment/article is posted/edited/deleted, the *entire* cache directory will be cleared via: "foreach(glob('cache/*') as $f) unlink($f);". Is this efficient? My limited knowledge of filesystems tells me that unlink should be efficient because all it does is sever the link between the name and file, and then the data is removed whenever possible. But unfortunately that is my very limited knowledge of the subject.

I could make cache removal very selective, but it seems like more work than its worth. Any tips?

Reply RSS

Replies

append delete #1. Johann

"I could make cache removal very selective, but it seems like more work than its worth. Any tips?"

Dunno if that helps, but here's how I do it:

1.) At the start of every request, generate a hash from the combination of

- user permissions (a bitfield of stuff like "is admin", "is profile owner" etc. )
- GET and POST (used to cache searches)
- configuration cookie

in short, every little thing that has an impact on the output returned needs to be in that hash.

2.) Check if the current request is already cached, return cached result if that is the case. Otherwise:

3.) Register all "items relevant to the cache". If you list a folder for example, any change to that folder, any change to a file, deleting or adding a file will be registered as "relevant" to that cache. For now, the relevant items get stored in an array. If the current requests actually gets cached (has to be turned on explicitly, the default is to not cache), the cache as well as the "relevant items" get stored in the database.

db_cache
hash
content
time_created
time_expire

db_cache_relevant_item
hash
type
id

where "type" is things like "add page to profile", "add subpage to parent page", and "id" is the id of the profile, the parent page etc. that "type" relates to.

4.) In all appropriate places (editing a page, deleting a file, etc.), sprinkle in a function call to delete cached requests that have this item registered as relevant.

cache->invalidate($type, $id)
{
'DELETE cache as c, cache_relevant_item as cri FROM cache as c, cache_relevant_item as cri WHERE c.hash = cri.hash AND cri.type = "' . $type . '" AND cri.id = '.$id
}

The amount of relevant item types and the calls to cache->register_relevant_item() and cache->invalidate() can mushroom quite a bit.... there are more elegant ways to do it, I'm sure.

And of course better ways to explain it ^^ however, it's the best I came up with so far, and it does work very nicely, at least for now.

append delete #2. JJ

Thanks Johann, I think I have the action of caching down. (BTW, if you're caching POST requests, wouldn't you have to throw the entire raw request body into the hash? Just seems kind of strange hehe). However, I don't want to have to deal with a DB for my cache. Part of the reason I'm caching pages is so I don't have to access a DB at all. I want to keep it as simple as possible, so I think I'd actually rather stick with my clear all the cache method since the weblog's content isn't *that* dynamic (for example there aren't any users logged in etc.). I'm just concerned whether clearing the entire cache directory whenever something changes is efficient. I'm curious about how fast it would be performance-wise with regards to the filesystem.

append delete #3. Dan Zollman

First off, I don't know the answer to your question.

Second, I have a similar caching mechanism, but with selective removal. It works well because of the way my blog system is structured; I don't know whether it would be a good idea to generalize this for other sites, but maybe it's worth sharing.

There are three parts to the process:

1. In the cache directory, there is a folder for each URI minus the GET parameters. For example, there is folder for article.php. Inside that folder, there is a cache for every combination of GET params: article.php?foo=1, article.php?foo=2, etc.

2. When a given page is changed, the page's cache folder can be deleted. When a comment is posted on article.php, then the article.php cache folder is deleted. Even if there were multiple versions from different GET params, that takes care of all of them.

3. Each function that affects page content has a set of instructions or a pattern that describes what cache items need to be removed. When a new article is posted or updated, the cache for every article listing page is cleared.

I only delete the entire cache when I'm pushing an update to the site itself.

append delete #4. JJ

Thanks for the advice, Dan. I might consider something like that. I just worry about how complex it might end up being. If I am going to do it, I would want to make sure I've found a really elegant way to implement it.

append delete #5. Johann

"BTW, if you're caching POST requests, wouldn't you have to throw the entire raw request body into the hash?"

Well yeah, that is, I just json_encode() the $_POST array and append that to the input for the hash. It's a bit overkill because it only ever ends up being used for the search, but it's the only way I could see to allow for that.

"Part of the reason I'm caching pages is so I don't have to access a DB at all."

Sure, but in my case the startup time (include() and init) comes close to the cost of the MySQL connection, so I figured to not care about those 10-15 miliseconds and enjoy all the convenience doing that in the database offers :)

Not to hijack the thread though (at least not until your question has been answered :D)

append delete #6. JJ

Johann, I've been thinking of caching POST requests and just dumping file_get_contents('php://input') into the hash unconditionally so I don't have to check for them, but it seems unnecessary for my weblog system which doesn't have any meaningful output for POST requests.

Anway, thanks for your suggestions guys. After stumbling over it for a few days, I *finally* found an elegant solution. Instead of clearing the entire cache directory when SQL is executed, I compare the filemtime of the potential cache file to the filemtime of my .sqlite file. If the cache is older than the .sqlite file, ignore it, serve the page as normal and cache it before outputting it. This way, it only deletes (or overwrites) the old cache files "on demand" so to speak. They only get deleted (or overwritten I should say) when the specific URI the cache file corresponds to is requested. This is also better because now if my .sqlite database is altered or changed by something outside of the weblog system (like an sqlite db manager or something), it will know not to serve the old cached files.

Finally got it the way I like it. Thanks guys.

append delete #7. JJ

Actually - another filesystem question now that I think about it, if anyone is willing: If two users visit the site and end up overwriting the same cache file at the same time, isn't there a possibility the both of them writing to the file at the same time could corrupt the data? Is it wise to use flock() or LOCK_EX in the file_put_contents call? Again, I'm not an expert when it comes to filesystems, so I apologize if this is a stupid question.

append delete #8. Kroc

Yes, use locking.*

*locking is not 100% perfect: apenwarr.ca/… In super-high-volumes sites, try caching to the DB. For most purposes, it will do.

append delete #9. JJ

That article looks pretty thorough. Thanks kroc.

#10. chiry

This post was deleted by a moderator

append delete #11. sull

"Is it wise to use flock() or LOCK_EX in the file_put_contents call?"

is a good decision to use it. i also like to use designated folders for users and other objects whenever feasible so that their are no occurrences of two+ users ever writing to the same files. you could also do this for things like comment threads on an object and recursively construct the thread along with periodic/random cache file creation with logic to check time modified (file system data or file metadata) to determine which data to serve (and post-processes for cache updating) etc etc.

append delete #12. debug

test

append delete #13. Kroc

Please don’t do that. If you’re going to post, say something. I can assure you it all works, we’ve been using the forum for months :P

Reply

(Leave this as-is, it’s a trap!)

There is no need to “register”, just enter the same name + password of your choice every time.

Pro tip: Use markup to add links, quotes and more.

Your friendly neighbourhood moderators: Kroc, Impressed, Martijn